2024-03-29T11:03:39.696 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:03:39.697 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680 --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/orig.config.yaml 2024-03-29T11:03:39.733 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:03:39.815 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622680' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCErejqCMDIglx+W5lAHopXR137IR6ag7yNhJsmQPKTHquPBzHI5AkhyWOyvIJqKqh/X9ekU9S3OZ6seNYT841M= smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPRTi96wnTvLKRcksy5oYbWtPhrckGCVWAYOJ5Z/+oJPF+AQexN0e4F3hF+M1K7g2ueVfNWA1GoFJluCEqjwyU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:03:39.815 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:03:39.815 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:03:39.816 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:03:39.817 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:03:39.841 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:03:39.841 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:03:39.841 INFO:teuthology.packaging:ref: None 2024-03-29T11:03:39.841 INFO:teuthology.packaging:tag: None 2024-03-29T11:03:39.842 INFO:teuthology.packaging:branch: main 2024-03-29T11:03:39.842 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:03:39.842 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:03:40.097 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:03:40.099 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:03:40.107 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:03:40.107 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:03:40.116 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:03:40.130 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:03:40.136 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:03:40.161 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:59:02.104971', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCErejqCMDIglx+W5lAHopXR137IR6ag7yNhJsmQPKTHquPBzHI5AkhyWOyvIJqKqh/X9ekU9S3OZ6seNYT841M='} 2024-03-29T11:03:40.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:59:02.106100', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPRTi96wnTvLKRcksy5oYbWtPhrckGCVWAYOJ5Z/+oJPF+AQexN0e4F3hF+M1K7g2ueVfNWA1GoFJluCEqjwyU='} 2024-03-29T11:03:40.180 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:03:40.188 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:03:40.188 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:03:40.188 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:03:40.266 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe2d86d20d0>, signals=[15]) 2024-03-29T11:03:40.266 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:03:40.275 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:03:40.275 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:03:40.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:03:40.386 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:03:40.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:03:40.491 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:03:40.498 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-29T11:03:40.517 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-29T11:03:40.518 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-29T11:03:40.574 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-29T11:03:40.574 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-29T11:03:40.574 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:03:40.575 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:03:40.576 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-29T11:03:40.599 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-03-29T11:03:40.617 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-03-29T11:03:40.617 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:03:40.676 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:03:40.677 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:03:40.677 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:03:40.677 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:03:40.677 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-03-29T11:03:40.698 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:03:40.708 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:03:40.714 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:03:40.714 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:03:40.716 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:03:40.733 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:03:40.739 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:03:40.739 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:03:40.773 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:03:40.803 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:03:40.859 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:03:40.874 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:03:40.874 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:03:40.874 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:03:40.875 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:03:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:03:40.891 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-29T11:03:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:03:40.892 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-29T11:03:40.948 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:03:40.949 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-29T11:03:40.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-29T11:03:40.950 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:03:40.951 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-361.el9.x86_64 2024-03-29T11:03:40.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-03-29T11:03:43.089 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:03:43.089 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:03:44.531 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:03:44.531 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:03:45.430 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-29T11:03:47.148 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-29T11:03:52.174 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-29T11:03:53.529 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 951 kB/s | 8.0 MB 00:08 2024-03-29T11:03:54.008 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-29T11:03:55.017 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-29T11:03:57.454 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 6.7 MB/s | 19 MB 00:02 2024-03-29T11:04:00.276 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-29T11:04:01.897 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 7.8 MB/s | 6.1 MB 00:00 2024-03-29T11:04:02.616 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:04:02.616 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:04:02.717 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:04:02.726 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:04:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:04:02.726 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:04:02.727 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:04:02.728 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-29T11:04:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:02.728 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-29T11:04:02.728 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-29T11:04:02.728 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:04:02.877 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:04:03.352 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T11:04:03.531 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-29T11:04:04.210 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-03-29T11:04:04.486 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T11:04:04.489 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:04:04.489 INFO:teuthology.orchestra.run.smithi003.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-29T11:04:04.489 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:04:04.921 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:04:04.921 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:04:05.792 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:04:05.792 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:04:06.541 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 7.2 MB/s | 21 MB 00:02 2024-03-29T11:04:08.200 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:04:08.827 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:04:08.921 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:04:11.766 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 51 kB/s | 1.7 kB 00:00 2024-03-29T11:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:04:14.139 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:04:14.140 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:04:14.240 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:04:14.248 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:04:14.249 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 76 M 2024-03-29T11:04:14.250 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 123 M 2024-03-29T11:04:14.250 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:04:14.442 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 996 kB/s | 192 kB 00:00 2024-03-29T11:04:14.893 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-29T11:04:15.418 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 29 MB/s | 34 MB 00:01 2024-03-29T11:04:15.784 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 21 MB/s | 28 MB 00:01 2024-03-29T11:04:15.787 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:04:15.787 INFO:teuthology.orchestra.run.smithi032.stdout:Total 50 MB/s | 76 MB 00:01 2024-03-29T11:04:15.788 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:04:16.197 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:04:16.197 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:04:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:04:17.038 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:17.064 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:04:17.065 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:04:19.538 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:04:20.151 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:04:20.244 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:04:22.367 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:23.128 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:04:23.264 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:04:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:04:28.424 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:33.550 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:53.066 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:04:53.066 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:53.111 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:04:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:04:53.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:04:53.867 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:53.868 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:04:54.225 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:04:54.324 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:04:54.324 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:04:54.324 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:04:54.325 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:04:54.325 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:04:54.325 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:04:54.325 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:04:54.325 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:04:54.408 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:04:54.409 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:04:54.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-29T11:04:54.456 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:04:54.457 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:04:54.458 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:04:54.458 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:04:54.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:04:55.036 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-29T11:04:56.506 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:04:56.526 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-29T11:04:56.528 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-29T11:04:56.546 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.1jZGi8PJBS 2024-03-29T11:04:56.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1jZGi8PJBS 2024-03-29T11:04:56.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.1jZGi8PJBS 2024-03-29T11:04:56.764 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.1jZGi8PJBS is 6KB 2024-03-29T11:04:56.814 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.1jZGi8PJBS 2024-03-29T11:04:56.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:04:56.897 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:04:56.897 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:04:56.897 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:04:56.897 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:04:56.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:04:57.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-29T11:05:04.366 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:05:04.367 INFO:teuthology.orchestra.run.smithi032.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:05:04.367 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:05:04.411 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:05:04.821 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:05:04.821 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:05:04.821 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:05:04.821 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:05:05.239 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:05:05.240 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:05:05.560 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:05:05.714 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.715 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:05:05.715 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:05:05.716 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:05:05.716 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:05:05.716 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:05:05.716 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:05:05.716 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:05:05.799 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:05:05.799 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:05:05.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-03-29T11:05:05.844 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-03-29T11:05:05.844 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-03-29T11:05:05.844 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-03-29T11:05:05.844 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-03-29T11:05:05.844 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:05:05.845 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:05:05.846 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:05:05.847 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:05:05.847 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:05:05.847 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:05:06.478 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-03-29T11:05:07.857 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:05:07.901 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-03-29T11:05:07.903 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-29T11:05:07.918 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.WUQmw8xH2s 2024-03-29T11:05:07.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WUQmw8xH2s 2024-03-29T11:05:07.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.WUQmw8xH2s 2024-03-29T11:05:08.128 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.WUQmw8xH2s is 6KB 2024-03-29T11:05:08.197 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.WUQmw8xH2s 2024-03-29T11:05:08.219 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:05:08.306 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:05:08.306 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:05:08.306 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:05:08.306 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:05:08.315 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:05:08.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-03-29T11:05:27.064 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:05:27.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:05:27.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:38.490 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:05:38.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:05:38.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:45.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-03-29T11:05:54.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:05:54.512 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:05:57.022 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-03-29T11:06:06.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:06:06.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:24.415 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-03-29T11:06:36.427 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:06:36.428 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:36.938 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-29T11:06:37.476 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:06:37.476 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:06:37.477 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-29T11:06:37.490 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:06:37.491 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:06:37.491 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:06:37.491 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:06:38.491 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:06:38.492 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:06:38.578 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-03-29T11:06:38.612 DEBUG:teuthology.parallel:result is None 2024-03-29T11:06:54.514 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:07:06.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:07:06.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:07:06.784 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-29T11:07:07.160 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:07:07.160 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:07:07.160 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-29T11:07:07.176 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:07:07.176 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:07:07.176 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:07:07.176 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:07:08.177 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:07:08.177 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:07:08.241 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-29T11:07:08.273 DEBUG:teuthology.parallel:result is None 2024-03-29T11:07:08.273 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:07:08.284 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:07:08.284 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:07:08.286 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:07:08.305 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:07:08.316 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:07:08.329 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:07:08.329 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:07:08.341 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:07:08.373 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:07:08.384 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:07:08.384 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:07:08.405 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:07:08.444 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:07:08.449 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:07:08.459 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:07:08.463 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:07:08.464 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:07:08.474 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:07:08.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:07:08.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:07:08.534 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:07:08.546 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:07:08.547 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:07:08.549 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:07:08.589 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:07:08.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:07:08.697 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:07:08.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:07:08.822 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:07:08.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:07:08.890 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:07:08.929 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:07:08.986 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:07:09.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:07:09.112 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:07:09.113 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:07:09.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-29T11:07:09.184 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-03-29T11:07:09.233 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:07:09.275 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:07:09.740 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:07:09.751 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:07:09.751 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:07:09.777 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:07:09.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-29T11:07:09.846 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-29T11:07:09.848 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-03-29T11:07:09.910 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-03-29T11:07:09.912 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:07:09.912 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-29T11:07:09.936 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-29T11:07:09.936 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-03-29T11:07:09.958 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-03-29T11:07:09.958 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi032.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:07:09.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:07:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:10.002 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:07:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:10.032 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:07:10.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:07:10.080 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:07:10.110 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:07:10.120 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T11:07:10.127 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T11:07:10.304 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:07:10.320 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:07:10.322 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi032.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:11:41.675 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi032.front.sepia.ceph.com')] 2024-03-29T11:11:41.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:11:41.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:41.783 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-29T11:11:41.869 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-29T11:11:41.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:11:41.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:41.965 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-29T11:11:42.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-29T11:11:42.050 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:11:42.061 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:11:42.062 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:11:42.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:11:42.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:11:42.065 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:11:42.096 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:11:42.113 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:11:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:11:42.139 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:11:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-29T11:11:42.158 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-29T11:11:42.169 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-03-29T11:11:42.175 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:11:42.185 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-03-29T11:11:42.191 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:11:42.199 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:11:42.215 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:11:42.242 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:11:42.246 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:11:42.246 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-29T11:11:42.246 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.246 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.247 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.247 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.269 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:11:42.272 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:11:42.273 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-29T11:11:42.273 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.273 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.273 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.273 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:11:42.273 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:11:42.282 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:11:42.282 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:11:42.282 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:11:42.282 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:11:42.288 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:11:42.288 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:11:42.289 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:11:42.289 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:42.291 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:11:42.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:11:42.437 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:11:42.438 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:11:42.450 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:11:42.450 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:11:42.557 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:11:42.557 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:11:42.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:11:42.566 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:11:42.566 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:11:42.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:11:42.585 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:11:42.586 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:11:42.597 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:11:42.597 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:11:42.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:11:42.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:11:42.735 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:11:42.746 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:11:42.802 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-03-29T11:11:42.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-03-29T11:11:42.812 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-03-29T11:11:42.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-03-29T11:11:43.055 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-03-29T11:11:43.059 INFO:teuthology.orchestra.run.smithi032.stdout:74 files removed 2024-03-29T11:11:43.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-03-29T11:11:43.086 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-03-29T11:11:43.705 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 259 kB/s | 82 kB 00:00 2024-03-29T11:11:43.757 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 225 kB/s | 82 kB 00:00 2024-03-29T11:11:44.082 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T11:11:44.183 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-29T11:11:44.489 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2024-03-29T11:11:44.533 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-29T11:11:45.011 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T11:11:45.166 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T11:11:46.595 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:11:46.625 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-29T11:11:47.577 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:11:47.674 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:11:51.192 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 8.0 MB/s | 19 MB 00:02 2024-03-29T11:11:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 19 MB 00:02 2024-03-29T11:11:55.734 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 6.1 MB 00:00 2024-03-29T11:11:55.820 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 6.1 MB 00:00 2024-03-29T11:11:57.330 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-29T11:11:57.477 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T11:11:59.078 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:12:00.089 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-03-29T11:12:04.262 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-29T11:12:05.231 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-29T11:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout:Install 30 Packages 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-03-29T11:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 58 M 2024-03-29T11:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:12:07.956 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:12:07.970 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:07.970 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:12:07.970 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:07.970 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:12:07.970 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:12:07.971 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:12:07.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:12:07.973 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:07.974 INFO:teuthology.orchestra.run.smithi032.stdout:Install 30 Packages 2024-03-29T11:12:07.974 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-03-29T11:12:07.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:07.974 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 58 M 2024-03-29T11:12:07.975 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:12:08.846 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 3.5 MB/s | 5.3 MB 00:01 2024-03-29T11:12:09.071 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 11 MB/s | 20 MB 00:01 2024-03-29T11:12:09.096 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 100 kB/s | 25 kB 00:00 2024-03-29T11:12:09.155 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-29T11:12:09.239 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.4 MB/s | 498 kB 00:00 2024-03-29T11:12:09.356 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 386 kB/s | 45 kB 00:00 2024-03-29T11:12:09.540 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 9.7 MB 00:02 2024-03-29T11:12:09.565 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 630 kB/s | 131 kB 00:00 2024-03-29T11:12:09.632 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-29T11:12:09.657 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.4 MB/s | 162 kB 00:00 2024-03-29T11:12:09.682 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-rados-19.0.0-1439.g5dcb190d.el 2.7 MB/s | 320 kB 00:00 2024-03-29T11:12:09.707 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.9 MB/s | 298 kB 00:00 2024-03-29T11:12:09.732 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 99 kB 00:00 2024-03-29T11:12:09.866 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 576 kB/s | 106 kB 00:00 2024-03-29T11:12:09.891 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 866 kB/s | 159 kB 00:00 2024-03-29T11:12:09.908 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 356 kB/s | 44 kB 00:00 2024-03-29T11:12:09.982 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 5.3 MB/s | 9.7 MB 00:01 2024-03-29T11:12:10.000 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-29T11:12:10.017 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 381 kB/s | 41 kB 00:00 2024-03-29T11:12:10.024 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 608 kB/s | 25 kB 00:00 2024-03-29T11:12:10.042 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-29T11:12:10.059 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2024-03-29T11:12:10.076 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 597 kB/s | 35 kB 00:00 2024-03-29T11:12:10.093 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:12:10.124 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.2 MB/s | 742 kB 00:00 2024-03-29T11:12:10.183 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 8.4 MB/s | 498 kB 00:00 2024-03-29T11:12:10.333 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.4 MB/s | 5.3 MB 00:02 2024-03-29T11:12:10.376 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 92 kB/s | 26 kB 00:00 2024-03-29T11:12:10.392 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 775 kB/s | 45 kB 00:00 2024-03-29T11:12:10.475 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:12:10.485 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-29T11:12:10.552 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 649 kB/s | 309 kB 00:00 2024-03-29T11:12:10.559 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 14 MB/s | 5.1 MB 00:00 2024-03-29T11:12:10.578 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-29T11:12:10.584 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.5 MB/s | 162 kB 00:00 2024-03-29T11:12:10.627 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T11:12:10.652 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.3 MB/s | 298 kB 00:00 2024-03-29T11:12:10.677 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T11:12:10.737 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T11:12:10.770 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-29T11:12:10.820 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 635 kB/s | 106 kB 00:00 2024-03-29T11:12:10.844 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 953 kB/s | 159 kB 00:00 2024-03-29T11:12:10.862 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-03-29T11:12:10.870 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-29T11:12:10.920 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-29T11:12:10.945 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T11:12:10.970 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-03-29T11:12:10.987 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-29T11:12:10.996 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:12:11.021 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 699 kB/s | 35 kB 00:00 2024-03-29T11:12:11.055 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 791 kB/s | 46 kB 00:00 2024-03-29T11:12:11.345 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:12:11.472 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 20 MB 00:03 2024-03-29T11:12:11.520 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 4.5 MB/s | 3.4 MB 00:00 2024-03-29T11:12:11.527 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:11.527 INFO:teuthology.orchestra.run.smithi003.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T11:12:11.527 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:12:11.567 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:12:11.567 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:12:11.838 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 71 kB/s | 26 kB 00:00 2024-03-29T11:12:11.897 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 353 kB/s | 309 kB 00:00 2024-03-29T11:12:12.014 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 277 kB/s | 49 kB 00:00 2024-03-29T11:12:12.056 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2024-03-29T11:12:12.067 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:12:12.068 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:12:12.148 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-03-29T11:12:12.298 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T11:12:12.323 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 717 kB/s | 192 kB 00:00 2024-03-29T11:12:12.399 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-29T11:12:12.632 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 10 MB/s | 3.1 MB 00:00 2024-03-29T11:12:12.974 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-03-29T11:12:12.982 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:12.982 INFO:teuthology.orchestra.run.smithi032.stdout:Total 12 MB/s | 58 MB 00:05 2024-03-29T11:12:12.982 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:12:13.025 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:12:13.025 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:12:13.521 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:12:13.521 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:12:13.564 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:12:13.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:12:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:12:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:12:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:12:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:12:13.908 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:12:13.972 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:12:14.318 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:12:14.400 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:14.495 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:12:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:12:14.683 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:12:14.712 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:12:14.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:12:14.826 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:12:14.848 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:12:14.912 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:12:14.946 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:12:15.015 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:12:15.056 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:12:15.093 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:12:15.098 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:12:15.121 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:12:15.138 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:12:15.146 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:12:15.169 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:12:15.170 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:12:15.186 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:12:15.200 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:12:15.233 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:12:15.377 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:12:15.434 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:12:15.556 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:12:15.587 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:12:15.677 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:12:15.708 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:12:15.750 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:12:15.779 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:12:15.831 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:15.906 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:15.996 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:12:16.024 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:12:16.078 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:12:16.107 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:12:16.151 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:12:16.221 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:12:16.224 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:12:16.243 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:12:16.248 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:12:16.290 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:12:16.319 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:12:16.325 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:12:16.389 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:12:16.425 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:12:16.436 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:12:16.476 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:12:16.517 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:12:16.558 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:12:16.587 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:12:16.620 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:12:16.944 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:12:16.975 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:12:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:12:17.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:12:17.147 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:12:17.628 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:12:17.652 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:12:17.715 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:12:17.819 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:12:17.846 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:12:18.593 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:18.630 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:19.214 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:19.250 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:12:19.324 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:12:19.324 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:12:19.324 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:12:19.324 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:19.359 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:20.029 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:20.068 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:20.649 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:12:20.687 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:12:20.778 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:12:20.778 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:12:20.779 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:12:20.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:20.822 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:29.288 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:29.288 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-03-29T11:12:29.288 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-03-29T11:12:29.289 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:30.213 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:12:30.237 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:12:30.237 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:30.237 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:12:30.238 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:12:30.238 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:12:30.238 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:30.239 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:12:30.324 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:12:30.324 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-03-29T11:12:30.782 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-03-29T11:12:30.783 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-03-29T11:12:30.783 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:31.494 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:12:31.495 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:12:31.496 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:12:31.497 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:12:31.497 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:12:31.497 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:12:31.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:12:31.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:31.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:12:31.498 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:12:31.710 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:12:31.736 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:31.738 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:12:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:12:31.821 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:32.502 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:32.502 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.503 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:12:32.504 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:32.505 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:12:32.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-03-29T11:12:32.920 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:12:32.921 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:12:32.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:12:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:12:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:12:32.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:12:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:12:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:12:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:12:32.924 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:12:33.382 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:12:33.547 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:12:33.548 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 48 M 2024-03-29T11:12:33.549 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 204 M 2024-03-29T11:12:33.550 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:12:33.806 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T11:12:33.848 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-29T11:12:33.881 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:12:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:12:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:33.956 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-03-29T11:12:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.956 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.957 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.958 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:33.959 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:12:33.983 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 454 kB/s | 163 kB 00:00 2024-03-29T11:12:34.082 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-29T11:12:34.285 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-03-29T11:12:34.852 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:12:35.011 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:12:35.012 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:35.012 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:12:35.012 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:35.012 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:12:35.012 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-03-29T11:12:35.013 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:35.014 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 48 M 2024-03-29T11:12:35.014 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 204 M 2024-03-29T11:12:35.014 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:12:35.309 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:12:35.359 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-29T11:12:35.392 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:12:35.576 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 381 kB/s | 163 kB 00:00 2024-03-29T11:12:35.990 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 20 MB/s | 47 MB 00:02 2024-03-29T11:12:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:35.994 INFO:teuthology.orchestra.run.smithi003.stdout:Total 20 MB/s | 48 MB 00:02 2024-03-29T11:12:36.009 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:12:36.032 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:12:36.032 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:12:36.126 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 255 kB/s | 187 kB 00:00 2024-03-29T11:12:36.426 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:12:36.426 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:12:37.021 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:12:37.092 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:12:37.128 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:12:37.169 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:12:37.210 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:12:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:12:37.743 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T11:12:37.746 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:37.747 INFO:teuthology.orchestra.run.smithi032.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-29T11:12:37.763 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:12:37.785 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:12:37.785 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:12:38.182 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:12:38.182 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:12:38.777 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:12:38.853 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:12:38.899 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:12:38.931 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:12:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:12:38.996 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:12:41.969 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:12:42.044 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:12:43.718 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:12:43.806 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:12:44.255 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:12:44.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:12:44.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:12:44.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:12:44.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:12:44.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:44.634 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:12:44.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-03-29T11:12:45.473 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:12:45.641 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:12:45.644 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:12:45.645 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:12:45.646 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout:Install 36 Packages 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:45.647 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 M 2024-03-29T11:12:45.648 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 107 M 2024-03-29T11:12:45.648 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:12:46.092 INFO:teuthology.orchestra.run.smithi003.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:12:46.161 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:12:46.525 INFO:teuthology.orchestra.run.smithi003.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-29T11:12:46.543 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:12:46.543 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:46.543 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:46.544 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:12:46.575 INFO:teuthology.orchestra.run.smithi003.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 2.1 MB 00:00 2024-03-29T11:12:46.617 INFO:teuthology.orchestra.run.smithi003.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.1 MB/s | 1.5 MB 00:00 2024-03-29T11:12:46.676 INFO:teuthology.orchestra.run.smithi003.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T11:12:46.743 INFO:teuthology.orchestra.run.smithi003.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 144 kB 00:00 2024-03-29T11:12:46.768 INFO:teuthology.orchestra.run.smithi003.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:12:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 633 kB/s | 32 kB 00:00 2024-03-29T11:12:46.819 INFO:teuthology.orchestra.run.smithi003.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-29T11:12:46.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-03-29T11:12:46.844 INFO:teuthology.orchestra.run.smithi003.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-29T11:12:46.870 INFO:teuthology.orchestra.run.smithi003.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:12:46.920 INFO:teuthology.orchestra.run.smithi003.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:12:46.945 INFO:teuthology.orchestra.run.smithi003.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T11:12:46.970 INFO:teuthology.orchestra.run.smithi003.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:12:46.996 INFO:teuthology.orchestra.run.smithi003.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:12:47.163 INFO:teuthology.orchestra.run.smithi003.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 134 kB/s | 22 kB 00:00 2024-03-29T11:12:47.338 INFO:teuthology.orchestra.run.smithi003.stdout:(17/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 21 MB/s | 17 MB 00:00 2024-03-29T11:12:47.404 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:12:47.574 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:12:47.577 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:12:47.578 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:12:47.579 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout:Install 36 Packages 2024-03-29T11:12:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:47.581 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 30 M 2024-03-29T11:12:47.581 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 107 M 2024-03-29T11:12:47.581 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:12:47.780 INFO:teuthology.orchestra.run.smithi003.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 111 kB/s | 43 kB 00:00 2024-03-29T11:12:47.830 INFO:teuthology.orchestra.run.smithi003.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 226 kB/s | 151 kB 00:00 2024-03-29T11:12:47.880 INFO:teuthology.orchestra.run.smithi003.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 137 kB/s | 125 kB 00:00 2024-03-29T11:12:48.047 INFO:teuthology.orchestra.run.smithi003.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-03-29T11:12:48.072 INFO:teuthology.orchestra.run.smithi003.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-29T11:12:48.147 INFO:teuthology.orchestra.run.smithi003.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-29T11:12:48.165 INFO:teuthology.orchestra.run.smithi003.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 255 kB/s | 23 kB 00:00 2024-03-29T11:12:48.181 INFO:teuthology.orchestra.run.smithi003.stdout:(25/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 36 kB/s | 11 kB 00:00 2024-03-29T11:12:48.240 INFO:teuthology.orchestra.run.smithi003.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-29T11:12:48.257 INFO:teuthology.orchestra.run.smithi003.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-29T11:12:48.332 INFO:teuthology.orchestra.run.smithi032.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 27 kB/s | 6.6 kB 00:00 2024-03-29T11:12:48.357 INFO:teuthology.orchestra.run.smithi003.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 675 kB/s | 79 kB 00:00 2024-03-29T11:12:48.376 INFO:teuthology.orchestra.run.smithi003.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 241 kB/s | 46 kB 00:00 2024-03-29T11:12:48.399 INFO:teuthology.orchestra.run.smithi003.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-03-29T11:12:48.449 INFO:teuthology.orchestra.run.smithi003.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 178 kB/s | 16 kB 00:00 2024-03-29T11:12:48.465 INFO:teuthology.orchestra.run.smithi032.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-03-29T11:12:48.491 INFO:teuthology.orchestra.run.smithi003.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-29T11:12:48.575 INFO:teuthology.orchestra.run.smithi003.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 448 kB/s | 90 kB 00:00 2024-03-29T11:12:48.608 INFO:teuthology.orchestra.run.smithi003.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-29T11:12:48.649 INFO:teuthology.orchestra.run.smithi032.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-29T11:12:48.650 INFO:teuthology.orchestra.run.smithi003.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-29T11:12:48.675 INFO:teuthology.orchestra.run.smithi003.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-29T11:12:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:48.682 INFO:teuthology.orchestra.run.smithi003.stdout:Total 9.8 MB/s | 30 MB 00:03 2024-03-29T11:12:48.706 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:12:48.753 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:12:48.753 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:12:48.765 INFO:teuthology.orchestra.run.smithi032.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.1 MB/s | 247 kB 00:00 2024-03-29T11:12:48.916 INFO:teuthology.orchestra.run.smithi032.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T11:12:48.983 INFO:teuthology.orchestra.run.smithi032.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-29T11:12:49.017 INFO:teuthology.orchestra.run.smithi032.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 952 kB/s | 32 kB 00:00 2024-03-29T11:12:49.050 INFO:teuthology.orchestra.run.smithi032.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:12:49.092 INFO:teuthology.orchestra.run.smithi032.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 926 kB/s | 39 kB 00:00 2024-03-29T11:12:49.142 INFO:teuthology.orchestra.run.smithi032.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:12:49.161 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:12:49.161 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:12:49.209 INFO:teuthology.orchestra.run.smithi032.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:12:49.385 INFO:teuthology.orchestra.run.smithi032.stdout:(12/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 18 MB/s | 17 MB 00:00 2024-03-29T11:12:49.410 INFO:teuthology.orchestra.run.smithi032.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 514 kB/s | 103 kB 00:00 2024-03-29T11:12:49.427 INFO:teuthology.orchestra.run.smithi032.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:12:49.456 INFO:teuthology.orchestra.run.smithi032.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-29T11:12:49.536 INFO:teuthology.orchestra.run.smithi032.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.9 MB/s | 4.7 MB 00:01 2024-03-29T11:12:49.603 INFO:teuthology.orchestra.run.smithi032.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2024-03-29T11:12:49.655 INFO:teuthology.orchestra.run.smithi032.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 554 kB/s | 125 kB 00:00 2024-03-29T11:12:50.053 INFO:teuthology.orchestra.run.smithi032.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 96 kB/s | 43 kB 00:00 2024-03-29T11:12:50.187 INFO:teuthology.orchestra.run.smithi032.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 232 kB/s | 151 kB 00:00 2024-03-29T11:12:50.295 INFO:teuthology.orchestra.run.smithi032.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-03-29T11:12:50.321 INFO:teuthology.orchestra.run.smithi032.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 268 kB/s | 172 kB 00:00 2024-03-29T11:12:50.412 INFO:teuthology.orchestra.run.smithi032.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-29T11:12:50.438 INFO:teuthology.orchestra.run.smithi032.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-29T11:12:50.463 INFO:teuthology.orchestra.run.smithi032.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 876 kB/s | 358 kB 00:00 2024-03-29T11:12:50.522 INFO:teuthology.orchestra.run.smithi032.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-29T11:12:50.547 INFO:teuthology.orchestra.run.smithi032.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-03-29T11:12:50.572 INFO:teuthology.orchestra.run.smithi032.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-03-29T11:12:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:12:50.656 INFO:teuthology.orchestra.run.smithi032.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 589 kB/s | 79 kB 00:00 2024-03-29T11:12:50.673 INFO:teuthology.orchestra.run.smithi032.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 163 kB/s | 16 kB 00:00 2024-03-29T11:12:50.679 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:12:50.720 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:12:50.764 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:12:50.765 INFO:teuthology.orchestra.run.smithi032.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.2 MB/s | 272 kB 00:00 2024-03-29T11:12:50.782 INFO:teuthology.orchestra.run.smithi032.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-03-29T11:12:50.807 INFO:teuthology.orchestra.run.smithi032.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 596 kB/s | 90 kB 00:00 2024-03-29T11:12:50.829 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:12:50.872 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:12:50.899 INFO:teuthology.orchestra.run.smithi032.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T11:12:50.904 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:12:50.915 INFO:teuthology.orchestra.run.smithi032.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-29T11:12:50.958 INFO:teuthology.orchestra.run.smithi032.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-29T11:12:50.963 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:12:50.963 INFO:teuthology.orchestra.run.smithi032.stdout:Total 8.8 MB/s | 30 MB 00:03 2024-03-29T11:12:50.987 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:12:51.000 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:12:51.034 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:12:51.034 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:12:51.077 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:12:51.124 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:12:51.199 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:12:51.258 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:12:51.296 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:12:51.354 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:12:51.410 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:12:51.450 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:12:51.450 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:12:51.605 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:12:51.658 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:12:51.697 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:12:51.731 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:12:51.761 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:12:51.840 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:12:51.871 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:12:51.904 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:12:51.937 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:12:51.980 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:12:52.014 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:12:52.109 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:12:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:12:52.221 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:12:52.361 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:12:52.496 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:12:52.668 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:12:52.699 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:12:52.723 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:12:52.724 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:52.724 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:12:52.724 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:12:52.724 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:12:52.724 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:12:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:12:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:12:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:12:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:12:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:12:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:12:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:12:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:12:53.272 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:12:53.348 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:12:53.399 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:12:53.436 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:12:53.494 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:12:53.550 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:12:53.754 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:12:53.798 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:12:53.837 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:12:53.880 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:12:53.910 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:12:53.988 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:12:54.028 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:12:54.061 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:12:54.102 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:12:54.146 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:12:54.188 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:12:54.253 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:12:54.281 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:54.282 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:12:54.319 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:12:54.394 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:12:54.537 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:12:54.676 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:12:54.746 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:12:54.769 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:54.842 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:12:54.873 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:12:54.899 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:54.988 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:12:55.010 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:12:55.011 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:55.011 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:12:55.011 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:12:55.011 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:12:55.011 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:55.079 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:12:56.407 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:12:56.433 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:12:56.434 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:56.434 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:12:56.434 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:12:56.434 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:12:56.435 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:12:56.780 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:12:56.781 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:12:56.782 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:12:56.900 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:12:56.924 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:12:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:12:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:12:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:12:56.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:57.137 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:12:57.156 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:12:57.157 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:12:57.157 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:12:57.157 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:12:57.157 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:12:57.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:57.226 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:12:57.842 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:12:57.843 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:12:57.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:12:57.845 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:12:57.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:57.845 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:12:58.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-03-29T11:12:58.783 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:12:58.885 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:12:58.921 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:12:58.922 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:12:58.923 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:12:58.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:12:58.940 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:12:58.941 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:12:58.941 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:12:59.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-03-29T11:12:59.581 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:12:59.738 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:12:59.738 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:59.738 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 747 k 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 754 k 2024-03-29T11:12:59.739 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:12:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:12:59.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:12:59.978 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:12:59.979 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:12:59.980 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:00.133 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.9 MB/s | 747 kB 00:00 2024-03-29T11:13:00.134 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:00.134 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.8 MB/s | 747 kB 00:00 2024-03-29T11:13:00.134 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:00.140 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:00.140 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:00.146 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:00.146 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:00.223 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:00.316 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-03-29T11:13:00.658 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:00.741 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:00.893 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:00.995 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:01.050 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:01.052 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:01.052 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:01.111 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:01.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:01.398 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:01.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:13:01.687 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:01.846 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:01.851 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:01.851 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:01.856 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:01.856 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:01.856 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:13:01.856 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 747 k 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 754 k 2024-03-29T11:13:01.857 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:02.130 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T11:13:02.132 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:02.132 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T11:13:02.132 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:02.137 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:02.137 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:02.143 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:02.143 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:02.152 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:02.215 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:02.311 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:02.311 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:02.311 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-29T11:13:02.311 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:02.311 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:02.312 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 145 k 2024-03-29T11:13:02.313 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 436 k 2024-03-29T11:13:02.313 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:02.664 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:02.680 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 393 kB/s | 145 kB 00:00 2024-03-29T11:13:02.681 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:02.681 INFO:teuthology.orchestra.run.smithi003.stdout:Total 392 kB/s | 145 kB 00:00 2024-03-29T11:13:02.681 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:02.690 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:02.690 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:02.745 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:02.746 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:02.754 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:02.890 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:02.956 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:02.981 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:02.982 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:13:02.982 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:03.139 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:03.413 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:03.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:03.991 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:04.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-03-29T11:13:04.240 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:04.396 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:04.397 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:04.398 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:04.398 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 145 k 2024-03-29T11:13:04.398 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 436 k 2024-03-29T11:13:04.398 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:04.753 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 410 kB/s | 145 kB 00:00 2024-03-29T11:13:04.753 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:04.753 INFO:teuthology.orchestra.run.smithi032.stdout:Total 408 kB/s | 145 kB 00:00 2024-03-29T11:13:04.753 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:04.762 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:04.762 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:04.787 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:04.817 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:04.889 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:04.945 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:04.946 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:04.946 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:04.964 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:05.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:13:05.021 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:05.046 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:05.047 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:05.047 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:13:05.047 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:05.579 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:05.737 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:05.738 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-29T11:13:05.738 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:05.738 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:13:05.739 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-29T11:13:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-29T11:13:05.740 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:05.740 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.9 M 2024-03-29T11:13:05.741 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-29T11:13:05.741 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:06.071 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:13:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:06.072 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:06.072 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:06.072 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:06.192 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 118 kB/s | 31 kB 00:00 2024-03-29T11:13:06.218 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 53 kB/s | 15 kB 00:00 2024-03-29T11:13:06.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-03-29T11:13:06.351 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-29T11:13:06.385 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-29T11:13:06.477 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.2 MB/s | 1.8 MB 00:00 2024-03-29T11:13:06.728 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-03-29T11:13:06.874 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:13:06.892 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:06.944 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.0 MB/s | 144 kB 00:00 2024-03-29T11:13:06.995 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:07.011 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T11:13:07.052 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:07.053 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:07.054 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:07.078 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.8 MB/s | 2.0 MB 00:00 2024-03-29T11:13:07.103 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-29T11:13:07.110 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:13:07.681 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:07.837 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:13:07.838 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:07.839 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.9 M 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-03-29T11:13:07.840 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:08.053 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 507 kB/s | 863 kB 00:01 2024-03-29T11:13:08.061 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:08.061 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.0 MB/s | 6.9 MB 00:02 2024-03-29T11:13:08.081 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:08.102 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:08.102 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:08.231 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 169 kB/s | 31 kB 00:00 2024-03-29T11:13:08.288 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:08.288 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:08.322 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 55 kB/s | 15 kB 00:00 2024-03-29T11:13:08.398 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-29T11:13:08.440 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:13:08.606 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.2 MB/s | 1.8 MB 00:00 2024-03-29T11:13:08.646 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-03-29T11:13:08.875 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-29T11:13:08.926 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:08.966 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T11:13:08.997 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:13:09.032 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:13:09.033 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T11:13:09.067 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 750 kB/s | 144 kB 00:00 2024-03-29T11:13:09.088 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:13:09.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:13:09.117 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-29T11:13:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:13:09.251 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.3 MB/s | 2.0 MB 00:00 2024-03-29T11:13:09.259 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:09.259 INFO:teuthology.orchestra.run.smithi032.stdout:Total 4.8 MB/s | 6.9 MB 00:01 2024-03-29T11:13:09.275 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:09.293 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:09.293 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:09.348 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:13:09.440 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:13:09.470 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:09.470 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:09.488 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:13:09.515 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:13:09.547 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:13:09.803 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:13:09.847 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:13:09.993 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:10.076 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:13:10.120 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:13:10.167 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:13:10.205 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:13:10.312 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:13:10.444 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:13:10.537 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:13:10.585 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:13:10.612 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:13:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:13:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:13:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:13:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:13:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:13:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:13:10.644 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:13:10.901 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:13:10.950 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:13:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:10.999 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:11.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:13:11.745 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:13:11.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:13:11.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:13:11.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:13:11.746 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:13:11.811 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:11.969 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:13:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-29T11:13:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:11.973 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2024-03-29T11:13:11.973 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2024-03-29T11:13:11.973 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:12.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:13:12.089 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:13:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:13:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:13:12.090 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:12.090 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:12.335 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:13:12.542 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 212 kB/s | 32 kB 00:00 2024-03-29T11:13:12.575 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 523 kB/s | 17 kB 00:00 2024-03-29T11:13:12.617 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 837 kB/s | 34 kB 00:00 2024-03-29T11:13:12.676 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:13:12.877 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:13:12.915 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:07 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:12.935 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:13:12.960 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-29T11:13:13.069 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:13.070 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-29T11:13:13.070 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:13.070 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-29T11:13:13.070 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:13.070 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:13:13.071 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 37 M 2024-03-29T11:13:13.072 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 209 M 2024-03-29T11:13:13.073 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:13.095 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-14 11 MB/s | 7.4 MB 00:00 2024-03-29T11:13:13.203 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 216 kB/s | 32 kB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 346 kB/s | 17 kB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 458 kB/s | 34 kB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.9 MB/s | 3.0 MB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-14 11 MB/s | 7.4 MB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-29T11:13:14.418 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 904 kB/s | 217 kB 00:00 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 195 kB 00:00 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 760 kB/s | 245 kB 00:00 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:Total 24 MB/s | 37 MB 00:01 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:14.419 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:14.420 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:14.560 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 675 kB/s | 195 kB 00:00 2024-03-29T11:13:14.726 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:14.734 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-29T11:13:14.759 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-29T11:13:14.792 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T11:13:14.793 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:13:14.794 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:14.794 INFO:teuthology.orchestra.run.smithi032.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T11:13:14.832 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:13:14.834 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:14.859 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:14.859 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:14.934 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:13:14.969 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:13:15.152 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:13:15.185 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:13:15.233 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:15.233 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:15.269 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:13:15.948 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:15.953 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:13:16.033 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:13:16.060 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:13:16.072 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:13:16.172 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:13:16.217 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:13:16.402 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:13:16.432 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:13:16.514 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:13:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:13:17.316 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:13:17.374 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:13:18.444 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:13:18.475 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:13:18.632 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:13:19.702 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:13:19.731 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:13:20.932 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:13:20.933 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:13:20.933 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:13:20.933 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:13:21.372 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:13:21.372 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:21.372 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:21.372 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:21.372 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:21.373 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:21.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:13:22.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:13:22.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:13:22.305 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:22.474 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:22.475 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:22.475 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:22.475 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:22.475 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:13:22.476 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-29T11:13:22.477 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:13:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:22.714 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:22.714 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:22.715 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:22.757 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2024-03-29T11:13:22.783 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 24 kB 00:00 2024-03-29T11:13:22.832 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:13:22.857 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:13:22.883 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 210 kB/s | 52 kB 00:00 2024-03-29T11:13:22.908 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T11:13:22.933 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-29T11:13:23.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:13:23.167 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-29T11:13:23.192 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-29T11:13:23.400 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 434 kB/s | 203 kB 00:00 2024-03-29T11:13:23.426 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 252 kB/s | 59 kB 00:00 2024-03-29T11:13:23.568 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 536 kB/s | 90 kB 00:00 2024-03-29T11:13:23.611 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:18 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:23.693 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T11:13:23.695 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:23.695 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T11:13:23.696 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:23.710 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:23.711 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:23.770 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:23.771 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:23.771 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:23.771 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:23.771 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:13:23.772 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.0 M 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-03-29T11:13:23.773 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:23.807 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:23.807 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:24.256 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2024-03-29T11:13:24.281 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 124 kB/s | 18 kB 00:00 2024-03-29T11:13:24.330 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:24.332 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:13:24.358 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:13:24.383 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:13:24.408 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2024-03-29T11:13:24.441 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 171 kB/s | 52 kB 00:00 2024-03-29T11:13:24.497 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:13:24.500 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-03-29T11:13:24.525 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2024-03-29T11:13:24.554 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:13:24.611 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:13:24.617 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-29T11:13:24.643 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-03-29T11:13:24.654 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:13:24.693 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-29T11:13:24.695 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:13:24.760 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T11:13:24.761 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:24.762 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-03-29T11:13:24.762 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:24.781 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:24.781 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:24.792 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:13:24.879 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:24.879 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:24.903 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:13:24.956 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:13:25.408 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:25.581 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:13:25.645 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:13:25.702 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:13:25.729 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:13:25.737 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:13:25.778 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:13:25.790 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:13:25.819 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:13:25.867 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:13:25.876 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:13:25.896 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:13:25.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:13:26.047 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:13:26.690 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:13:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:13:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:13:26.691 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:13:26.811 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:13:26.879 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:13:26.910 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:13:26.959 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:13:26.980 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:13:27.055 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:13:27.056 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:13:27.057 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:13:27.057 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:13:27.057 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:13:27.057 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:27.057 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:27.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:13:27.693 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:13:27.890 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:28.046 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:13:28.046 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:28.046 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:28.047 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:28.049 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:28.049 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:28.050 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-29T11:13:28.051 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-29T11:13:28.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:28.051 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2024-03-29T11:13:28.051 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-03-29T11:13:28.051 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:28.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:13:28.821 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T11:13:28.868 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:28.880 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 753 kB/s | 248 kB 00:00 2024-03-29T11:13:28.930 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 363 kB/s | 138 kB 00:00 2024-03-29T11:13:29.026 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:13:29.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:29.028 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.8 M 2024-03-29T11:13:29.029 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 33 M 2024-03-29T11:13:29.029 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:29.222 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 169 kB/s | 58 kB 00:00 2024-03-29T11:13:29.272 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-29T11:13:29.414 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 925 kB/s | 548 kB 00:00 2024-03-29T11:13:29.416 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:29.417 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-29T11:13:29.431 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:29.436 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:13:29.441 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:29.441 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:29.503 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 607 kB/s | 138 kB 00:00 2024-03-29T11:13:29.535 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:29.536 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:29.578 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-29T11:13:29.670 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 350 kB/s | 58 kB 00:00 2024-03-29T11:13:29.695 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 437 kB/s | 51 kB 00:00 2024-03-29T11:13:29.762 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-03-29T11:13:29.764 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:29.764 INFO:teuthology.orchestra.run.smithi032.stdout:Total 9.2 MB/s | 6.8 MB 00:00 2024-03-29T11:13:29.778 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:29.789 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:29.789 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:29.831 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:29.881 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:29.882 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:30.002 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:13:30.049 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:13:30.183 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:30.361 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:13:30.423 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:13:30.601 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:13:30.695 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:13:30.755 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:13:30.786 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:13:30.976 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:13:31.062 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:13:31.143 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:13:31.176 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:13:31.632 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:13:31.918 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:13:31.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:13:31.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:13:31.919 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:13:31.919 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:31.919 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:31.987 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:13:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:13:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:13:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:13:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:13:31.988 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:13:32.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-03-29T11:13:32.285 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:13:32.285 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:32.285 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:32.286 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:32.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-03-29T11:13:32.716 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:32.874 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:32.875 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-29T11:13:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:32.876 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 964 k 2024-03-29T11:13:32.876 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.8 M 2024-03-29T11:13:32.876 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:33.015 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 570 kB/s | 79 kB 00:00 2024-03-29T11:13:33.093 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:33.249 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:33.249 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:33.249 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:33.249 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:33.249 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-03-29T11:13:33.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:33.251 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 964 k 2024-03-29T11:13:33.251 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.8 M 2024-03-29T11:13:33.251 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:33.331 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2024-03-29T11:13:33.332 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:33.333 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T11:13:33.333 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:33.355 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:33.355 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:33.372 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 653 kB/s | 79 kB 00:00 2024-03-29T11:13:33.433 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:33.433 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:33.547 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 885 kB 00:00 2024-03-29T11:13:33.548 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:33.548 INFO:teuthology.orchestra.run.smithi032.stdout:Total 3.2 MB/s | 964 kB 00:00 2024-03-29T11:13:33.549 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:33.570 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:33.571 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:33.620 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:33.646 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:33.647 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:13:33.831 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:13:33.977 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:13:34.051 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:13:34.526 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:13:34.526 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:13:34.771 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:13:34.771 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:13:34.826 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:13:34.827 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:34.827 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:34.827 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:13:34.827 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:34.827 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:35.060 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:35.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2024-03-29T11:13:35.317 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-volume 2024-03-29T11:13:35.683 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:35.844 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 332 k 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.5 M 2024-03-29T11:13:35.845 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:35.901 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:30 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:35.968 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 561 kB/s | 69 kB 00:00 2024-03-29T11:13:36.062 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:36.062 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:36.062 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-29T11:13:36.062 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 332 k 2024-03-29T11:13:36.063 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.5 M 2024-03-29T11:13:36.064 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:36.185 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 562 kB/s | 69 kB 00:00 2024-03-29T11:13:36.210 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 724 kB/s | 264 kB 00:00 2024-03-29T11:13:36.211 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:36.211 INFO:teuthology.orchestra.run.smithi003.stdout:Total 907 kB/s | 332 kB 00:00 2024-03-29T11:13:36.212 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:36.221 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:36.222 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:36.261 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:36.261 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:36.310 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T11:13:36.311 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:36.311 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-29T11:13:36.311 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:36.321 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:36.321 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:36.357 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:36.358 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:36.412 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:36.514 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:36.593 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:13:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:13:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:13:36.646 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:36.646 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:13:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:36.692 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:13:36.721 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:13:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:13:36.737 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:36.737 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:13:36.737 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:13:37.560 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:37.638 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:37.852 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:37.853 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:37.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-03-29T11:13:38.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-03-29T11:13:38.522 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:38.679 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:38.680 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:38.680 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-29T11:13:38.680 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:38.680 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2024-03-29T11:13:38.681 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:38.773 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:38.898 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 584 kB/s | 127 kB 00:00 2024-03-29T11:13:38.899 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:38.899 INFO:teuthology.orchestra.run.smithi003.stdout:Total 580 kB/s | 127 kB 00:00 2024-03-29T11:13:38.899 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:38.905 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:38.905 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:38.930 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:38.931 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:38.931 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-29T11:13:38.931 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:38.931 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:38.931 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 127 k 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 456 k 2024-03-29T11:13:38.932 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:38.963 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:38.963 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:39.109 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:39.217 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:39.290 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 354 kB/s | 127 kB 00:00 2024-03-29T11:13:39.290 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:39.290 INFO:teuthology.orchestra.run.smithi032.stdout:Total 353 kB/s | 127 kB 00:00 2024-03-29T11:13:39.290 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:39.296 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:39.296 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:39.355 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:39.355 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:39.501 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:39.668 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:39.710 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:39.994 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:39.994 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:39.994 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:39.994 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:39.994 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:39.995 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:40.137 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:40.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-03-29T11:13:40.452 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:40.453 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:40.453 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:40.453 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:40.453 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:40.453 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:40.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-03-29T11:13:40.803 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:40.906 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:40.963 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:40.964 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:40.964 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:41.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-03-29T11:13:41.273 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:41.374 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:41.432 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:41.433 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:41.433 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:41.501 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-03-29T11:13:41.603 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:41.760 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:41.761 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:41.761 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-29T11:13:41.761 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 140 k 2024-03-29T11:13:41.762 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:42.010 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 126 kB/s | 31 kB 00:00 2024-03-29T11:13:42.010 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:42.010 INFO:teuthology.orchestra.run.smithi003.stdout:Total 125 kB/s | 31 kB 00:00 2024-03-29T11:13:42.010 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:42.014 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:42.014 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:42.046 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:42.047 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:42.089 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:42.151 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:42.249 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:42.250 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 31 k 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 140 k 2024-03-29T11:13:42.251 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:42.261 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:42.420 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 184 kB/s | 31 kB 00:00 2024-03-29T11:13:42.421 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:42.421 INFO:teuthology.orchestra.run.smithi032.stdout:Total 183 kB/s | 31 kB 00:00 2024-03-29T11:13:42.421 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:42.424 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:42.424 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:42.446 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:42.447 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:42.576 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:42.645 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:42.678 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:42.918 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:43.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-03-29T11:13:43.077 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:43.334 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:43.492 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-03-29T11:13:43.655 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:43.756 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:43.810 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:43.811 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:43.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-03-29T11:13:44.069 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:44.169 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:44.223 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:44.224 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:44.224 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:44.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-03-29T11:13:44.435 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:44.533 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:44.588 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:44.588 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:44.589 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:44.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-03-29T11:13:44.848 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:44.949 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:45.004 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:45.005 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:45.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-03-29T11:13:45.214 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:45.321 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:45.372 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:45.373 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:45.373 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:45.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-03-29T11:13:45.635 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:45.741 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:45.798 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:45.800 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:45.800 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:45.859 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-03-29T11:13:45.997 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:46.103 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:46.154 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:46.155 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:46.155 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:46.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-03-29T11:13:46.439 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:46.538 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:46.593 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:46.594 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:46.594 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:46.643 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-03-29T11:13:46.778 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:46.879 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:46.933 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:46.934 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:46.934 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:46.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-03-29T11:13:47.225 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:47.326 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:47.381 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:47.382 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:47.382 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:47.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-03-29T11:13:47.559 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:47.662 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:47.717 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:47.718 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-29T11:13:47.718 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:47.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-03-29T11:13:48.008 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:48.109 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:13:48.166 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:48.167 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-29T11:13:48.167 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:48.228 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-03-29T11:13:48.343 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:48.497 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:48.497 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:48.497 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-29T11:13:48.497 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:48.498 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:48.499 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 85 k 2024-03-29T11:13:48.499 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 231 k 2024-03-29T11:13:48.499 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:48.704 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 416 kB/s | 85 kB 00:00 2024-03-29T11:13:48.704 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:48.704 INFO:teuthology.orchestra.run.smithi003.stdout:Total 414 kB/s | 85 kB 00:00 2024-03-29T11:13:48.704 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:48.712 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:48.712 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:48.770 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:48.770 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:48.800 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:48.913 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:48.954 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:48.955 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:48.955 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:48.956 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:48.957 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 85 k 2024-03-29T11:13:48.957 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 231 k 2024-03-29T11:13:48.957 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:49.005 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:49.265 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 276 kB/s | 85 kB 00:00 2024-03-29T11:13:49.265 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:49.265 INFO:teuthology.orchestra.run.smithi032.stdout:Total 275 kB/s | 85 kB 00:00 2024-03-29T11:13:49.266 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:49.273 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:49.331 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:49.332 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:49.394 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:49.473 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:49.576 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:49.695 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:49.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-03-29T11:13:49.967 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:50.259 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:50.259 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:50.260 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:50.486 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:50.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-03-29T11:13:50.644 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.1 M 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 13 M 2024-03-29T11:13:50.645 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:51.173 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-29T11:13:51.173 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:51.174 INFO:teuthology.orchestra.run.smithi003.stdout:Total 5.8 MB/s | 3.1 MB 00:00 2024-03-29T11:13:51.174 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:51.184 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:51.184 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:51.260 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:51.260 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:51.283 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:13:51.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.1 M 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 13 M 2024-03-29T11:13:51.285 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:51.711 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:51.798 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:51.798 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:51.798 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:13:51.798 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:13:51.799 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:13:51.799 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:51.818 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.8 MB/s | 3.1 MB 00:00 2024-03-29T11:13:51.818 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:51.819 INFO:teuthology.orchestra.run.smithi032.stdout:Total 5.8 MB/s | 3.1 MB 00:00 2024-03-29T11:13:51.819 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:51.829 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:51.830 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:51.907 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:51.907 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:13:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:52.890 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:52.890 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:52.890 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:52.891 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:52.891 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:53.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:53.567 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:53.679 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:12:04 AM UTC. 2024-03-29T11:13:53.796 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-03-29T11:13:53.836 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:53.837 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 171 k 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 494 k 2024-03-29T11:13:53.838 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-29T11:13:54.184 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 496 kB/s | 171 kB 00:00 2024-03-29T11:13:54.184 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:54.185 INFO:teuthology.orchestra.run.smithi003.stdout:Total 494 kB/s | 171 kB 00:00 2024-03-29T11:13:54.185 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-29T11:13:54.193 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-29T11:13:54.193 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-29T11:13:54.252 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-29T11:13:54.253 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-29T11:13:54.372 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:12:05 AM UTC. 2024-03-29T11:13:54.407 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-29T11:13:54.519 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-29T11:13:54.527 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 171 k 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 494 k 2024-03-29T11:13:54.528 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-29T11:13:54.954 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 403 kB/s | 171 kB 00:00 2024-03-29T11:13:54.955 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:13:54.955 INFO:teuthology.orchestra.run.smithi032.stdout:Total 402 kB/s | 171 kB 00:00 2024-03-29T11:13:54.955 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-29T11:13:54.964 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-29T11:13:54.964 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-29T11:13:55.018 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:55.023 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-29T11:13:55.023 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-29T11:13:55.175 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-29T11:13:55.280 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:55.302 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-29T11:13:55.583 DEBUG:teuthology.parallel:result is None 2024-03-29T11:13:55.686 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:13:55.924 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:55.924 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-29T11:13:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:13:55.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:55.925 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-29T11:13:56.178 DEBUG:teuthology.parallel:result is None 2024-03-29T11:13:56.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:13:56.408 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:13:56.430 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:13:56.430 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:13:56.431 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:13:56.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:13:56.656 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:13:56.678 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:13:56.678 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:13:56.678 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:13:56.680 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:13:56.680 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:56.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:13:56.711 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:56.712 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:13:56.747 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:13:56.748 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:56.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:13:56.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:13:56.849 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:56.849 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:13:56.878 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:13:56.947 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:13:56.948 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:56.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:13:56.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:13:57.043 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:57.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:13:57.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:13:57.140 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:13:57.140 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:57.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:13:57.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:13:57.237 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:57.237 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:13:57.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:13:57.333 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:13:57.434 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:13:57.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /var/log/ceph 2024-03-29T11:13:57.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-03-29T11:13:57.465 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:13:57.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:13:57.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:13:57.533 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:13:57.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:13:57.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:13:57.616 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:13:57.616 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:13:57.616 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622680', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi003.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCErejqCMDIglx+W5lAHopXR137IR6ag7yNhJsmQPKTHquPBzHI5AkhyWOyvIJqKqh/X9ekU9S3OZ6seNYT841M=', 'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPRTi96wnTvLKRcksy5oYbWtPhrckGCVWAYOJ5Z/+oJPF+AQexN0e4F3hF+M1K7g2ueVfNWA1GoFJluCEqjwyU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:13:57.617 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:13:57.645 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:13:57.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:13:57.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:13:57.728 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:57.728 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:13:57.782 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:13:57.782 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-03-29T11:13:57.837 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-29 11:13:52.060392609 +0000 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-29 11:11:14.038185221 +0000 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-29 11:11:14.038185221 +0000 2024-03-29T11:13:57.838 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-29 11:11:14.038185221 +0000 2024-03-29T11:13:57.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:13:57.904 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-29T11:13:57.904 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-29T11:13:57.904 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.00019975 s, 2.6 MB/s 2024-03-29T11:13:57.906 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:13:57.961 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-03-29T11:13:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:13:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.016 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-29 11:13:52.061392595 +0000 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-29 11:11:14.386180190 +0000 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-29 11:11:14.386180190 +0000 2024-03-29T11:13:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-29 11:11:14.386180190 +0000 2024-03-29T11:13:58.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:13:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-29T11:13:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-29T11:13:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000179335 s, 2.9 MB/s 2024-03-29T11:13:58.083 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:13:58.139 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-29 11:13:52.061392595 +0000 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-29 11:11:14.726175273 +0000 2024-03-29T11:13:58.194 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-29 11:11:14.726175273 +0000 2024-03-29T11:13:58.195 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-29 11:11:14.726175273 +0000 2024-03-29T11:13:58.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:13:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-29T11:13:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-29T11:13:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000186295 s, 2.7 MB/s 2024-03-29T11:13:58.261 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:13:58.317 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-29 11:13:52.061392595 +0000 2024-03-29T11:13:58.372 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-29 11:11:15.091169993 +0000 2024-03-29T11:13:58.373 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-29 11:11:15.091169993 +0000 2024-03-29T11:13:58.373 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-29 11:11:15.091169993 +0000 2024-03-29T11:13:58.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:13:58.437 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-29T11:13:58.437 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-29T11:13:58.438 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000190526 s, 2.7 MB/s 2024-03-29T11:13:58.438 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:13:58.494 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:13:58.494 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:58.494 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:13:58.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:13:58.513 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-03-29T11:13:58.569 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:13:58.569 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.569 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-29 11:13:52.722518014 +0000 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-29 11:11:18.371445214 +0000 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-29 11:11:18.371445214 +0000 2024-03-29T11:13:58.570 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-29 11:11:18.371445214 +0000 2024-03-29T11:13:58.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:13:58.637 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-29T11:13:58.637 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-29T11:13:58.637 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000103242 s, 5.0 MB/s 2024-03-29T11:13:58.638 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:13:58.695 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-03-29T11:13:58.751 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:13:58.751 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.751 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:13:58.751 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.752 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.752 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-29 11:13:52.722518014 +0000 2024-03-29T11:13:58.752 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-29 11:11:18.716438186 +0000 2024-03-29T11:13:58.752 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-29 11:11:18.716438186 +0000 2024-03-29T11:13:58.752 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-29 11:11:18.716438186 +0000 2024-03-29T11:13:58.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:13:58.818 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-29T11:13:58.818 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-29T11:13:58.818 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.0001 s, 5.1 MB/s 2024-03-29T11:13:58.819 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:13:58.876 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-29 11:13:52.722518014 +0000 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-29 11:11:19.060431178 +0000 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-29 11:11:19.060431178 +0000 2024-03-29T11:13:58.935 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-29 11:11:19.060431178 +0000 2024-03-29T11:13:58.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:13:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-29T11:13:59.001 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-29T11:13:59.002 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000117452 s, 4.4 MB/s 2024-03-29T11:13:59.003 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:13:59.058 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-03-29T11:13:59.113 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-29 11:13:52.723517994 +0000 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-29 11:11:19.392424414 +0000 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-29 11:11:19.392424414 +0000 2024-03-29T11:13:59.114 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-29 11:11:19.392424414 +0000 2024-03-29T11:13:59.114 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:13:59.178 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-29T11:13:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-29T11:13:59.179 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 8.5542e-05 s, 6.0 MB/s 2024-03-29T11:13:59.180 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:13:59.236 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:13:59.236 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:13:59.236 INFO:tasks.ceph:Generating config... 2024-03-29T11:13:59.237 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:13:59.237 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:13:59.237 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:13:59.237 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:13:59.237 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:13:59.238 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:13:59.238 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:13:59.238 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:13:59.238 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:13:59.238 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:13:59.238 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:13:59.239 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:13:59.239 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:13:59.239 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:13:59.239 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:13:59.239 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:13:59.239 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:13:59.240 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:13:59.240 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:13:59.240 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:13:59.240 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:13:59.241 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:13:59.241 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:13:59.241 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:13:59.242 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:13:59.243 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:13:59.243 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:13:59.243 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:13:59.243 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:13:59.243 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:13:59.243 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:13:59.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:13:59.300 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:13:59.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:13:59.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:13:59.377 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.3:6789]'), ('mon.c', '[v1:172.21.15.3:6790]'), ('mon.b', '[v1:172.21.15.32:6789]')] 2024-03-29T11:13:59.378 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.32:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:13:59.379 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:59.379 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:13:59.432 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.3:6789]' --addv c '[v1:172.21.15.3:6790]' --addv b '[v1:172.21.15.32:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:13:59.517 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:13:59.517 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: generated fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 0 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:min_mon_release 17 (quincy) 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:election_strategy: 1 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:13:59.518 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:13:59.519 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:13:59.534 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bc4c2e6b-f00f-48b8-a384-90ebb3c19722... 2024-03-29T11:13:59.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:13:59.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:13:59.624 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-03-29T11:13:59.624 INFO:teuthology.orchestra.run.smithi003.stdout: chdir = "" 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: auth supported = cephx 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: filestore xattr use omap = true 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: mon clock drift allowed = 1.000 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: osd crush chooseleaf type = 0 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: auth debug = true 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on old message = true 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: ms die on bug = true 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.625 INFO:teuthology.orchestra.run.smithi003.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon pg warn max object skew = 0 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default size = 2 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow primary affinity = true 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd allow pg remap = true 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on no sortbitwise = false 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on too few osds = false 2024-03-29T11:13:59.626 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon_allow_pool_size_one = true 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: osd default data pool replay window = 5 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon allow pool delete = true 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon cluster log file level = debug 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: debug asserts on shutdown = true 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon health detail to clog = false 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.32:6789]" 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: lockdep = True 2024-03-29T11:13:59.627 INFO:teuthology.orchestra.run.smithi003.stdout: mon client directed command retry = 5 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: mon election default strategy = 1 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: ms bind msgr2 = False 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: ms inject socket failures = 5000 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: ms type = async 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout:[osd] 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: osd journal size = 100 2024-03-29T11:13:59.628 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub load threshold = 5.0 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd scrub max interval = 600 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd recover clone overlap = true 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd recovery max chunk = 1048576 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug shutdown = true 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug op order = true 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify stray on activate = true 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug trim objects = true 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.630 INFO:teuthology.orchestra.run.smithi003.stdout: osd open classes on start = true 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug pg log writeout = true 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: osd map max advance = 10 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: journal zero on create = true 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: filestore apply finisher threads = 3 2024-03-29T11:13:59.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.632 INFO:teuthology.orchestra.run.smithi003.stdout: bdev debug aio = true 2024-03-29T11:13:59.632 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug misdirected ops = true 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore block size = 96636764160 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression algorithm = zstd 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore compression mode = aggressive 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore fsck on mount = True 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: bluestore zero block detection = True 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluefs = 20 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug bluestore = 20 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug osd = 20 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: debug rocksdb = 10 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd full ratio = 0.9 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:13:59.633 INFO:teuthology.orchestra.run.smithi003.stdout: osd class default list = * 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd class load list = * 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd client watch timeout = 120 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify cached snaps = True 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd debug verify missing on start = True 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd mclock override recovery settings = True 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd objectstore = bluestore 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue = debug_random 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd op queue cut off = debug_random 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: osd shutdown pgref assert = True 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr] 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-29T11:13:59.634 INFO:teuthology.orchestra.run.smithi003.stdout: debug mgr = 20 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: mgr/telemetry/nag = false 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout:[mon] 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-29T11:13:59.635 INFO:teuthology.orchestra.run.smithi003.stdout: debug mon = 20 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: debug paxos = 20 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: debug auth = 20 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon data avail warn = 5 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon mgr mkfs grace = 240 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd reporter subtree level = osd 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon osd prime pg temp = true 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:13:59.636 INFO:teuthology.orchestra.run.smithi003.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: # 1m isn't quite enough 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon min osdmap epochs = 50 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune interval = 2 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune min = 15 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon scrub interval = 300 2024-03-29T11:13:59.637 INFO:teuthology.orchestra.run.smithi003.stdout: mon warn on pool no app = False 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: paxos service trim min = 10 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout:[client] 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: rgw cache enabled = true 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable ops log = true 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: rgw enable usage log = true 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: debug monc = 20 2024-03-29T11:13:59.638 INFO:teuthology.orchestra.run.smithi003.stdout: debug ms = 1 2024-03-29T11:13:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: debug objecter = 20 2024-03-29T11:13:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: debug rados = 20 2024-03-29T11:13:59.639 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.a] 2024-03-29T11:13:59.639 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.c] 2024-03-29T11:13:59.639 INFO:teuthology.orchestra.run.smithi003.stdout:[mon.b] 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-29T11:13:59.640 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.641 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-29T11:13:59.642 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-29T11:13:59.642 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:13:59.642 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:13:59.642 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-29T11:13:59.642 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.643 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "[v1:172.21.15.3:6789],[v1:172.21.15.3:6790],[v1:172.21.15.32:6789]" 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: lockdep = True 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: mon client directed command retry = 5 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: mon election default strategy = 1 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: ms bind msgr2 = False 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: ms inject socket failures = 5000 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: ms type = async 2024-03-29T11:13:59.644 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-29T11:13:59.645 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-29T11:13:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-29T11:13:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-29T11:13:59.647 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.648 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression algorithm = zstd 2024-03-29T11:13:59.654 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression mode = aggressive 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore zero block detection = True 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 20 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 20 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug monc = 20 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 10 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: osd class default list = * 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: osd class load list = * 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: osd client watch timeout = 120 2024-03-29T11:13:59.655 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify cached snaps = True 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify missing on start = True 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock override recovery settings = True 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd op queue = debug_random 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd op queue cut off = debug_random 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: osd shutdown pgref assert = True 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:13:59.656 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:13:59.657 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon min osdmap epochs = 50 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune interval = 2 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune min = 15 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon scrub interval = 300 2024-03-29T11:13:59.658 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on pool no app = False 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: paxos service trim min = 10 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: debug monc = 20 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: debug objecter = 20 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout: debug rados = 20 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-29T11:13:59.659 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-29T11:13:59.661 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:13:59.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:13:59.718 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:13:59.718 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:59.718 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:13:59.733 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:59.733 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:13:59.787 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:13:59.787 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:59.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:13:59.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:13:59.869 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:13:59.869 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:13:59.924 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:13:59.924 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:59.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:13:59.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:13:59.964 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:13:59.964 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:00.019 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:14:00.019 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:14:00.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:14:00.074 INFO:teuthology.orchestra.run.smithi003.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:14:00.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:14:00.148 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:14:00.150 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:14:00.151 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:14:00.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:14:00.194 INFO:teuthology.orchestra.run.smithi003.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:14:00.214 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:14:00.260 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:14:00.275 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:14:00.275 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi003.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:14:00.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:14:00.302 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:00.302 INFO:tasks.ceph:role: osd.0 2024-03-29T11:14:00.302 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:14:00.302 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:14:00.376 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:00.376 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:00.377 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:00.890 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-29T11:14:00.892 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-29T11:14:00.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:14:01.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:14:01.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:14:01.142 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:01.142 INFO:tasks.ceph:role: osd.1 2024-03-29T11:14:01.142 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:14:01.142 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:14:01.216 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:01.217 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:01.663 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-29T11:14:01.665 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-29T11:14:01.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:14:01.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:14:01.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:14:01.863 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:01.864 INFO:tasks.ceph:role: osd.2 2024-03-29T11:14:01.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:14:01.864 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:14:01.938 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:01.938 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:01.939 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:02.437 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-29T11:14:02.439 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-29T11:14:02.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:14:02.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:14:02.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:14:02.680 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:02.680 INFO:tasks.ceph:role: osd.3 2024-03-29T11:14:02.680 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:14:02.680 DEBUG:teuthology.orchestra.run.smithi003:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:14:02.755 INFO:teuthology.orchestra.run.smithi003.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:02.755 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:02.755 INFO:teuthology.orchestra.run.smithi003.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:02.755 INFO:teuthology.orchestra.run.smithi003.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:02.755 INFO:teuthology.orchestra.run.smithi003.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:02.756 INFO:teuthology.orchestra.run.smithi003.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:02.756 INFO:teuthology.orchestra.run.smithi003.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:02.756 INFO:teuthology.orchestra.run.smithi003.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:02.756 INFO:teuthology.orchestra.run.smithi003.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:02.756 INFO:teuthology.orchestra.run.smithi003.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:03.263 INFO:teuthology.orchestra.run.smithi003.stdout:Discarding blocks...Done. 2024-03-29T11:14:03.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi003.front.sepia.ceph.com -o noatime 2024-03-29T11:14:03.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:14:03.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:14:03.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.549+0000 7f5b61c13640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:14:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.550+0000 7f5b61c13640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:14:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.550+0000 7f5b61c13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:14:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.550+0000 7f5b61c13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:14:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.550+0000 7f5b61c13640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:14:03.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:03.550+0000 7f5b61c13640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:14:06.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:14:06.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:06.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.527+0000 7f78ddd2a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:14:06.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.528+0000 7f78ddd2a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:14:06.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.528+0000 7f78ddd2a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:14:06.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.528+0000 7f78ddd2a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:14:06.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.528+0000 7f78ddd2a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:14:06.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:06.528+0000 7f78ddd2a640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:14:09.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:14:09.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:09.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.497+0000 7f970d60e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:14:09.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.497+0000 7f970d60e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:14:09.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.497+0000 7f970d60e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:14:09.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.497+0000 7f970d60e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:14:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.497+0000 7f970d60e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:14:09.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:09.498+0000 7f970d60e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:14:12.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:14:12.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:14:12.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:12.480+0000 7f8cf28f5640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:14:15.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:14:15.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:14:15.371 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:15.371 INFO:tasks.ceph:role: osd.4 2024-03-29T11:14:15.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:14:15.372 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:14:15.442 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:15.443 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:15.475 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-29T11:14:15.477 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-29T11:14:15.477 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:14:15.550 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:14:15.623 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:14:15.692 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:15.693 INFO:tasks.ceph:role: osd.5 2024-03-29T11:14:15.693 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:14:15.693 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:15.760 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:15.761 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:15.761 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:15.792 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-29T11:14:15.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-29T11:14:15.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:14:15.875 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:14:15.947 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:14:16.015 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:16.015 INFO:tasks.ceph:role: osd.6 2024-03-29T11:14:16.015 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:14:16.015 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:16.081 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:16.113 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-29T11:14:16.115 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-29T11:14:16.115 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:14:16.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:14:16.261 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:14:16.329 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:14:16.329 INFO:tasks.ceph:role: osd.7 2024-03-29T11:14:16.329 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:14:16.329 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:14:16.395 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:14:16.395 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:14:16.395 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:14:16.395 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:14:16.395 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:14:16.396 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:14:16.396 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:14:16.396 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:14:16.396 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:14:16.396 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:14:16.427 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-29T11:14:16.429 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-29T11:14:16.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:14:16.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:14:16.573 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:14:16.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:16.701+0000 7efcec42a640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:14:19.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:14:19.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:14:19.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:14:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:14:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:14:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:14:19.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:19.670+0000 7f99a5af9640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:14:22.493 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:14:22.527 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.641+0000 7ff05ef19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.641+0000 7ff05ef19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.642+0000 7ff05ef19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.642+0000 7ff05ef19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.642+0000 7ff05ef19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:14:22.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:22.642+0000 7ff05ef19640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:14:25.464 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:14:25.495 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:25.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.608+0000 7f90338a6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:14:25.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.608+0000 7f90338a6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:14:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.608+0000 7f90338a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:14:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.608+0000 7f90338a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:14:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.608+0000 7f90338a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:14:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:25.609+0000 7f90338a6640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:14:28.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:14:28.459 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:14:28.459 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:28.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:14:28.492 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:28.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:14:28.562 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:28.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:14:28.631 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:28.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:14:28.700 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:28.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:14:28.769 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:28.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:14:28.801 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:28.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:14:28.872 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:28.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:14:28.943 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:28.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:14:29.018 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:29.019 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:14:29.113 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:29.113 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:14:29.134 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:29.134 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:14:29.173 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:14:29.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:14:29.179 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:14:29.224 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.y] 2024-03-29T11:14:29.224 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB4ogZmFvvRBRAAhQNgfd+Qf27GN+ETQ0meGA== 2024-03-29T11:14:29.224 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.0] 2024-03-29T11:14:29.224 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB7ogZmS+LTIBAAX3EcDaheweqKQDo83EDmeQ== 2024-03-29T11:14:29.225 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.1] 2024-03-29T11:14:29.225 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB+ogZmkIKDHxAASctyff1Suwo64fsUH6+HVQ== 2024-03-29T11:14:29.225 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.2] 2024-03-29T11:14:29.226 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCBogZm1cmyHRAAnMZfTGrQovNf68Ut07v9sg== 2024-03-29T11:14:29.226 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.3] 2024-03-29T11:14:29.226 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCEogZm0SeqHBAAzedhoXXM6v2PpYXdrPRE7g== 2024-03-29T11:14:29.227 INFO:teuthology.orchestra.run.smithi003.stdout:[mgr.x] 2024-03-29T11:14:29.227 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB4ogZm340+ChAABWLBlIn/gCdxbQa/L+OL3g== 2024-03-29T11:14:29.227 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.4] 2024-03-29T11:14:29.228 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCIogZmCHTZKRAAuG9yaU7+3jBK3Kl0maP4AA== 2024-03-29T11:14:29.228 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.5] 2024-03-29T11:14:29.228 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCLogZmDdkFKBAAdh8jROt4Lc9e6ZCzkMEFvg== 2024-03-29T11:14:29.228 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.6] 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCOogZmBiJRJhAAbeKopiFkJzO0qdRO8tJkpg== 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout:[osd.7] 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQCRogZmnqVTJBAAZTGYXeDR2rqN11zuhzK1hw== 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB4ogZmi0DqDBAAHBk7liHfewzN2vDM6Nz8pg== 2024-03-29T11:14:29.229 INFO:teuthology.orchestra.run.smithi003.stdout:[client.1] 2024-03-29T11:14:29.230 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQB4ogZmq5XoEBAASl104qVJxD1hgi3lbHvPJQ== 2024-03-29T11:14:29.244 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-03-29T11:14:29.244 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB4ogZmFvvRBRAAhQNgfd+Qf27GN+ETQ0meGA== 2024-03-29T11:14:29.244 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB7ogZmS+LTIBAAX3EcDaheweqKQDo83EDmeQ== 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB+ogZmkIKDHxAASctyff1Suwo64fsUH6+HVQ== 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCBogZm1cmyHRAAnMZfTGrQovNf68Ut07v9sg== 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCEogZm0SeqHBAAzedhoXXM6v2PpYXdrPRE7g== 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB4ogZm340+ChAABWLBlIn/gCdxbQa/L+OL3g== 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-03-29T11:14:29.245 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCIogZmCHTZKRAAuG9yaU7+3jBK3Kl0maP4AA== 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCLogZmDdkFKBAAdh8jROt4Lc9e6ZCzkMEFvg== 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCOogZmBiJRJhAAbeKopiFkJzO0qdRO8tJkpg== 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCRogZmnqVTJBAAZTGYXeDR2rqN11zuhzK1hw== 2024-03-29T11:14:29.246 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-29T11:14:29.247 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB4ogZmi0DqDBAAHBk7liHfewzN2vDM6Nz8pg== 2024-03-29T11:14:29.247 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2024-03-29T11:14:29.247 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQB4ogZmq5XoEBAASl104qVJxD1hgi3lbHvPJQ== 2024-03-29T11:14:29.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:14:29.278 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:14:29.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.417 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:14:29.579 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:14:29.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.801 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:14:29.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:14:29.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:14:29.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:14:29.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:14:29.973 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:14:29.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:14:30.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:14:30.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:14:30.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:14:30.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:14:30.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:14:30.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:14:30.342 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:14:30.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:14:30.468 DEBUG:teuthology.orchestra.run.smithi003:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:30.471 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:14:30.525 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:14:30.526 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:14:30.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:14:30.528 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:14:30.528 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:14:30.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:14:30.532 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:14:30.532 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:14:30.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:14:30.566 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:14:30.566 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:14:30.566 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:14:30.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:14:30.569 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:14:30.569 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:14:30.569 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:14:30.571 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:14:30.572 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:14:30.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.695+0000 7fe4009ba640 10 monclient: get_monmap_and_config 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.695+0000 7fe4009ba640 10 monclient: build_initial_monmap 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.695+0000 7fe4009ba640 10 monclient: monmap: 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:14:30.697594+0000 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:14:30.697594+0000 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:30.674 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:30.675 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:30.675 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:30.675 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:30.675 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:30.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.696+0000 7fe4009ba640 1 Processor -- start 2024-03-29T11:14:30.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.696+0000 7fe4009ba640 1 -- start start 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.696+0000 7fe4009ba640 10 monclient: init 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe3fc071f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe3fc0aeaf0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc0af520 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc0b0d50 con 0x7fe3fc0aeaf0 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc13fb80 con 0x7fe3fc0af520 2024-03-29T11:14:30.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc13fd10 con 0x7fe3fc071f20 2024-03-29T11:14:30.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:30.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe4009ba640 10 monclient(hunting): authenticate will time out at 797.952236s 2024-03-29T11:14:30.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe3fa59c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe3fc071f20 0x7fe3fc0b1170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:39314/0 (socket says 172.21.15.3:39314) 2024-03-29T11:14:30.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:30.697+0000 7fe3fa59c640 1 -- 172.21.15.3:0/1142770590 learned_addr learned my addr 172.21.15.3:0/1142770590 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:33.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): tick 2024-03-29T11:14:33.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 monclient(hunting): continuing hunt 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.3:6790/0 conn(0x7fe3fc071f20 legacy=0x7fe3fc0b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 legacy=0x7fe3fc13f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.3:6789/0 conn(0x7fe3fc0aeaf0 legacy=0x7fe3fc0b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): picked mon.noname-a con 0x7fe3dc0036e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc0af520 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): picked mon.noname-b con 0x7fe3fc071f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc151b30 con 0x7fe3dc0036e0 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc00f380 con 0x7fe3fc0af520 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3dc0105e0 con 0x7fe3fc071f20 2024-03-29T11:14:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:33.698+0000 7fe3ebfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679126923 0 0) 0x7fe3fc151b30 con 0x7fe3dc0036e0 2024-03-29T11:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3e4003520 con 0x7fe3dc0036e0 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370268595 0 0) 0x7fe3dc00f380 con 0x7fe3fc0af520 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc151b30 con 0x7fe3fc0af520 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f9d9b640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 legacy=0x7fe3dc009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f9d9b640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 legacy=0x7fe3dc009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f9d9b640 1 --1- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 0x7fe3dc009300 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f9d9b640 1 --1- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 0x7fe3dc009300 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:14:35.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.866+0000 7fe3f959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.32:6789/0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.868+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2779553451 0 0) 0x7fe3e4003520 con 0x7fe3dc0036e0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.3:6790/0 conn(0x7fe3fc071f20 legacy=0x7fe3dc00cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.32:6789/0 conn(0x7fe3fc0af520 legacy=0x7fe3dc009300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 monclient: found mon.noname-a 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3dc00e430 con 0x7fe3dc0036e0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: _finish_auth 0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe3ec0016d0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:35.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: dump: 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:35.848 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe4009ba640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe4009ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3ec0018a0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 1 -- 172.21.15.3:0/1142770590 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe3ec001c70 con 0x7fe3dc0036e0 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe3f959a640 10 monclient: dump: 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe4009ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:35.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.869+0000 7fe4009ba640 4 set_mon_vals no callback set 2024-03-29T11:14:35.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 10 monclient: shutdown 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 1 -- 172.21.15.3:0/1142770590 >> v1:172.21.15.3:6789/0 conn(0x7fe3dc0036e0 legacy=0x7fe3dc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 1 -- 172.21.15.3:0/1142770590 shutdown_connections 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 1 -- 172.21.15.3:0/1142770590 >> 172.21.15.3:0/1142770590 conn(0x7fe3fc0b7b60 msgr2=0x7fe3fc0b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 1 -- 172.21.15.3:0/1142770590 shutdown_connections 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.870+0000 7fe4009ba640 1 -- 172.21.15.3:0/1142770590 wait complete. 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.871+0000 7fe4009ba640 10 monclient: build_initial_monmap 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.871+0000 7fe4009ba640 10 monclient: monmap: 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:35.856 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 librados: starting msgr at 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 librados: starting objecter 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 Processor -- start 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 -- start start 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 librados: setting wanted keys 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 1 librados: calling monclient init 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.872+0000 7fe4009ba640 10 monclient: init 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-b con 0x7fe3dc0036e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:35.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-c con 0x7fe3fc071f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient(hunting): picked mon.noname-a con 0x7fe3fc0af520 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc06ff90 con 0x7fe3fc0af520 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc070270 con 0x7fe3dc0036e0 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc06e0d0 con 0x7fe3fc071f20 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe4009ba640 10 monclient(hunting): authenticate will time out at 803.127242s 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3fa59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe3dc0036e0 0x7fe3fc06e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:59698/0 (socket says 172.21.15.3:59698) 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3f9d9b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fe3fc071f20 0x7fe3fc06eb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47152/0 (socket says 172.21.15.3:47152) 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3fa59c640 1 -- 172.21.15.3:0/767308181 learned_addr learned my addr 172.21.15.3:0/767308181 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter ms_handle_connect 0x7fe3dc0036e0 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter ms_handle_connect 0x7fe3fc071f20 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:35.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter ms_handle_connect 0x7fe3fc0af520 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797890310 0 0) 0x7fe3fc06ff90 con 0x7fe3fc0af520 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.873+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3dc00ef20 con 0x7fe3fc0af520 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932136037 0 0) 0x7fe3fc070270 con 0x7fe3dc0036e0 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc06ff90 con 0x7fe3dc0036e0 2024-03-29T11:14:35.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156631552 0 0) 0x7fe3fc06e0d0 con 0x7fe3fc071f20 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3fc070270 con 0x7fe3fc071f20 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2385265636 0 0) 0x7fe3fc06ff90 con 0x7fe3dc0036e0 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3fc06e0d0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe3ec001bb0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): dump: 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.860 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1700735517 0 0) 0x7fe3fc06e0d0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 >> v1:172.21.15.3:6790/0 conn(0x7fe3fc071f20 legacy=0x7fe3fc06eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 >> v1:172.21.15.3:6789/0 conn(0x7fe3fc0af520 legacy=0x7fe3fc1e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 monclient: found mon.b 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:35.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc070690 con 0x7fe3dc0036e0 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe3eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe4009ba640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe4009ba640 10 monclient: _renew_subs 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe4009ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.875+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3fc070880 con 0x7fe3dc0036e0 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe4009ba640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:35.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe4009ba640 10 monclient: _renew_subs 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe4009ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3fc13bcf0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe4009ba640 1 librados: init done 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3ec001d50 con 0x7fe3dc0036e0 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe3ec004c70 con 0x7fe3dc0036e0 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:35.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 10 monclient: dump: 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe3ec005220 con 0x7fe3dc0036e0 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3f8d99640 4 set_mon_vals no callback set 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1987667977 0 0) 0x7fe3ec0065d0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 10 client.4114.objecter ms_dispatch 0x7fe3fc000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:14:35.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 3 client.4114.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 3 client.4114.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.876+0000 7fe3eaffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.877+0000 7fe4009ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.877+0000 7fe4009ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.877+0000 7fe4009ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.877+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e0005080 con 0x7fe3dc0036e0 2024-03-29T11:14:35.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.879+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe3ec006790 con 0x7fe3dc0036e0 2024-03-29T11:14:35.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.879+0000 7fe3eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:35.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.879+0000 7fe3eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.954+0000 7fe4009ba640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:14:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.954+0000 7fe4009ba640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:14:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.954+0000 7fe4009ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.954+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe3e0005330 con 0x7fe3dc0036e0 2024-03-29T11:14:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.957+0000 7fe3eaffd640 1 -- 172.21.15.3:0/767308181 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe3ec0055b0 con 0x7fe3dc0036e0 2024-03-29T11:14:35.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.957+0000 7fe3eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:14:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.957+0000 7fe3eaffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:14:35.935 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 10 librados: watch_flush enter 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 10 librados: watch_flush exit 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 10 monclient: shutdown 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 >> v1:172.21.15.32:6789/0 conn(0x7fe3dc0036e0 legacy=0x7fe3fc06e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 shutdown_connections 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 >> 172.21.15.3:0/767308181 conn(0x7fe3fc0b7b60 msgr2=0x7fe3fc12fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 shutdown_connections 2024-03-29T11:14:35.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 -- 172.21.15.3:0/767308181 wait complete. 2024-03-29T11:14:35.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:35.958+0000 7fe4009ba640 1 librados: shutdown 2024-03-29T11:14:35.947 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:14:35.947 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:14:35.948 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:35.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:14:35.975 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:35.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:14:36.042 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:36.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:14:36.110 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-29T11:14:36.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:14:36.178 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:36.178 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:14:36.204 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:36.204 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:14:36.271 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:36.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:14:36.339 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-29T11:14:36.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:14:36.344 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:36.366+0000 7f5c76038040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:14:36.345 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:36.368+0000 7f2eac012040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:14:36.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new 813cc1eb-fd67-4adc-9120-a9ef762b40d0 0 2024-03-29T11:14:36.400 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:36.422+0000 7f5c76038040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:14:36.403 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:36.425+0000 7f2eac012040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:14:36.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 10 monclient: get_monmap_and_config 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 10 monclient: monmap: 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:36.526280+0000 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:36.526280+0000 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:36.503 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 1 Processor -- start 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 1 -- start start 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.525+0000 7f5a42b23640 10 monclient: init 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3c0b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-a con 0x7f5a3c0a9ef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-c con 0x7f5a3c146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c145790 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0aabe0 con 0x7f5a3c146350 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0aaec0 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a42b23640 10 monclient(hunting): authenticate will time out at 799.789467s 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a410c0640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a3c146350 0x7f5a3c14da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34710/0 (socket says 172.21.15.32:34710) 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a410c0640 1 -- 172.21.15.32:0/3758740771 learned_addr learned my addr 172.21.15.32:0/3758740771 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.526+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651816917 0 0) 0x7f5a3c0aabe0 con 0x7f5a3c146350 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:14:36.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c003520 con 0x7f5a3c146350 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806646772 0 0) 0x7f5a3c0aaec0 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0aabe0 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763441500 0 0) 0x7f5a3c145790 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0aaec0 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266442035 0 0) 0x7f5a2c003520 con 0x7f5a3c146350 2024-03-29T11:14:36.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 >> v1:172.21.15.3:6790/0 conn(0x7f5a3c0b4a00 legacy=0x7f5a3c145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 >> v1:172.21.15.3:6789/0 conn(0x7f5a3c0a9ef0 legacy=0x7f5a3c0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c142170 con 0x7f5a3c146350 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a34001a10 con 0x7f5a3c146350 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:36.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: dump: 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.520 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a34001bf0 con 0x7f5a3c146350 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 4 set_mon_vals no callback set 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 1 -- 172.21.15.32:0/3758740771 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a34002040 con 0x7f5a3c146350 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a3b7fe640 10 monclient: dump: 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.521 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.522 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 10 monclient: shutdown 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 1 -- 172.21.15.32:0/3758740771 >> v1:172.21.15.32:6789/0 conn(0x7f5a3c146350 legacy=0x7f5a3c14da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 1 -- 172.21.15.32:0/3758740771 shutdown_connections 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.527+0000 7f5a42b23640 1 -- 172.21.15.32:0/3758740771 >> 172.21.15.32:0/3758740771 conn(0x7f5a3c066920 msgr2=0x7f5a3c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 1 -- 172.21.15.32:0/3758740771 shutdown_connections 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 1 -- 172.21.15.32:0/3758740771 wait complete. 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 10 monclient: monmap: 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:36.526 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 1 librados: starting msgr at 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 1 librados: starting objecter 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.528+0000 7f5a42b23640 1 Processor -- start 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 -- start start 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 librados: setting wanted keys 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 librados: calling monclient init 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient: init 2024-03-29T11:14:36.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-c con 0x7f5a3c0a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-a con 0x7f5a3c0b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient(hunting): picked mon.noname-b con 0x7f5a3c146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c13d5f0 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c0aff50 con 0x7f5a3c146350 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3c13d910 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a42b23640 10 monclient(hunting): authenticate will time out at 799.792638s 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a410c0640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a3c146350 0x7f5a3c13cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34726/0 (socket says 172.21.15.32:34726) 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a410c0640 1 -- 172.21.15.32:0/15900112 learned_addr learned my addr 172.21.15.32:0/15900112 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter ms_handle_connect 0x7f5a3c146350 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter ms_handle_connect 0x7f5a3c0a9ef0 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter ms_handle_connect 0x7f5a3c0b4a00 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.529+0000 7f5a397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722057532 0 0) 0x7f5a3c0aff50 con 0x7f5a3c146350 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a20004eb0 con 0x7f5a3c146350 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050566968 0 0) 0x7f5a3c13d910 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c0aff50 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767964447 0 0) 0x7f5a3c13d5f0 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a3c13d910 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005052690 0 0) 0x7f5a20004eb0 con 0x7f5a3c146350 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3c13d5f0 con 0x7f5a3c146350 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a34001f40 con 0x7f5a3c146350 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): dump: 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.531 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2071724807 0 0) 0x7f5a3c0aff50 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a20004eb0 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023577615 0 0) 0x7f5a3c13d910 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a3c0aff50 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a300018b0 con 0x7f5a3c0a9ef0 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): dump: 2024-03-29T11:14:36.532 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a28001770 con 0x7f5a3c0b4a00 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient(hunting): dump: 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3945213757 0 0) 0x7f5a3c13d5f0 con 0x7f5a3c146350 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 >> v1:172.21.15.3:6790/0 conn(0x7f5a3c0a9ef0 legacy=0x7f5a3c0b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 >> v1:172.21.15.3:6789/0 conn(0x7f5a3c0b4a00 legacy=0x7f5a3c13c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 monclient: found mon.b 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c1db220 con 0x7f5a3c146350 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 10 monclient: _renew_subs 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3c1db470 con 0x7f5a3c146350 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 10 monclient: _renew_subs 2024-03-29T11:14:36.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3c0afba0 con 0x7f5a3c146350 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a42b23640 1 librados: init done 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a34004e10 con 0x7f5a3c146350 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5a34005120 con 0x7f5a3c146350 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 monclient: dump: 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.536 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a3affd640 4 set_mon_vals no callback set 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5a34005610 con 0x7f5a3c146350 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 400371615 0 0) 0x7f5a34006a90 con 0x7f5a3c146350 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 10 client.4120.objecter ms_dispatch 0x7f5a3c000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.530+0000 7f5a397fa640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:14:36.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.531+0000 7f5a42b23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.531+0000 7f5a42b23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.531+0000 7f5a42b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.531+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c005080 con 0x7f5a3c146350 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.533+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5a34005a60 con 0x7f5a3c146350 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.533+0000 7f5a397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.533+0000 7f5a397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:36.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.607+0000 7f5a42b23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "813cc1eb-fd67-4adc-9120-a9ef762b40d0", "id": 0}] 2024-03-29T11:14:36.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.607+0000 7f5a42b23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "813cc1eb-fd67-4adc-9120-a9ef762b40d0", "id": 0}] 2024-03-29T11:14:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.607+0000 7f5a42b23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.607+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "813cc1eb-fd67-4adc-9120-a9ef762b40d0", "id": 0} v 0) v1 -- 0x7f5a1c0058b0 con 0x7f5a3c146350 2024-03-29T11:14:36.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.609+0000 7f5a397fa640 1 -- 172.21.15.32:0/15900112 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "813cc1eb-fd67-4adc-9120-a9ef762b40d0", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3782016450 0 732707515) 0x7f5a34005dd0 con 0x7f5a3c146350 2024-03-29T11:14:36.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.609+0000 7f5a397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "813cc1eb-fd67-4adc-9120-a9ef762b40d0", "id": 0}] 2024-03-29T11:14:36.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.609+0000 7f5a397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:36.588 INFO:teuthology.orchestra.run.smithi032.stdout:0 2024-03-29T11:14:36.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 10 librados: watch_flush enter 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 10 librados: watch_flush exit 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 10 monclient: shutdown 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 >> v1:172.21.15.32:6789/0 conn(0x7f5a3c146350 legacy=0x7f5a3c13cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 shutdown_connections 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 >> 172.21.15.32:0/15900112 conn(0x7f5a3c066920 msgr2=0x7f5a3c13e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 shutdown_connections 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 -- 172.21.15.32:0/15900112 wait complete. 2024-03-29T11:14:36.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.610+0000 7f5a42b23640 1 librados: shutdown 2024-03-29T11:14:36.599 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new 1cecc8b9-c752-4e32-9cd2-943602908be5 1 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 10 monclient: get_monmap_and_config 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 10 monclient: monmap: 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.709 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:36.722210+0000 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:36.722210+0000 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 1 Processor -- start 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 1 -- start start 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.721+0000 7fe9784f6640 10 monclient: init 2024-03-29T11:14:36.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9700b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9700a9ef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe970146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe970145790 con 0x7fe9700a9ef0 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700aabe0 con 0x7fe970146350 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700aaec0 con 0x7fe9700b4a00 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe9784f6640 10 monclient(hunting): authenticate will time out at 799.985417s 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe976a93640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe970146350 0x7fe97014da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34740/0 (socket says 172.21.15.32:34740) 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe976a93640 1 -- 172.21.15.32:0/512556978 learned_addr learned my addr 172.21.15.32:0/512556978 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102360515 0 0) 0x7fe9700aabe0 con 0x7fe970146350 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe95c003520 con 0x7fe970146350 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162138894 0 0) 0x7fe9700aaec0 con 0x7fe9700b4a00 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9700aabe0 con 0x7fe9700b4a00 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828351601 0 0) 0x7fe970145790 con 0x7fe9700a9ef0 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:14:36.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.722+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9700aaec0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3510094412 0 0) 0x7fe95c003520 con 0x7fe970146350 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 >> v1:172.21.15.3:6790/0 conn(0x7fe9700b4a00 legacy=0x7fe970145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 >> v1:172.21.15.3:6789/0 conn(0x7fe9700a9ef0 legacy=0x7fe9700b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 monclient: found mon.noname-c 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe970142170 con 0x7fe970146350 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe968001a10 con 0x7fe970146350 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: dump: 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe968001b70 con 0x7fe970146350 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 1 -- 172.21.15.32:0/512556978 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe968001f80 con 0x7fe970146350 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 4 set_mon_vals no callback set 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe975290640 10 monclient: dump: 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.715 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 10 monclient: shutdown 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 1 -- 172.21.15.32:0/512556978 >> v1:172.21.15.32:6789/0 conn(0x7fe970146350 legacy=0x7fe97014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 1 -- 172.21.15.32:0/512556978 shutdown_connections 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 1 -- 172.21.15.32:0/512556978 >> 172.21.15.32:0/512556978 conn(0x7fe970066920 msgr2=0x7fe970066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 1 -- 172.21.15.32:0/512556978 shutdown_connections 2024-03-29T11:14:36.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.723+0000 7fe9784f6640 1 -- 172.21.15.32:0/512556978 wait complete. 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 10 monclient: monmap: 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:36.717 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 librados: starting msgr at 2024-03-29T11:14:36.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 librados: starting objecter 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 Processor -- start 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 -- start start 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 librados: setting wanted keys 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 1 librados: calling monclient init 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.724+0000 7fe9784f6640 10 monclient: init 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9700a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9700b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe970146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9700afec0 con 0x7fe9700b4a00 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9701db2f0 con 0x7fe970146350 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9701da6c0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe9784f6640 10 monclient(hunting): authenticate will time out at 799.988319s 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe976a93640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe970146350 0x7fe9700b0b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34756/0 (socket says 172.21.15.32:34756) 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe976a93640 1 -- 172.21.15.32:0/3708652620 learned_addr learned my addr 172.21.15.32:0/3708652620 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter ms_handle_connect 0x7fe970146350 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter ms_handle_connect 0x7fe9700a9ef0 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter ms_handle_connect 0x7fe9700b4a00 2024-03-29T11:14:36.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048479728 0 0) 0x7fe9701db2f0 con 0x7fe970146350 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe954004eb0 con 0x7fe970146350 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853246607 0 0) 0x7fe9701da6c0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9701db2f0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552203869 0 0) 0x7fe9700afec0 con 0x7fe9700b4a00 2024-03-29T11:14:36.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9701da6c0 con 0x7fe9700b4a00 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098174299 0 0) 0x7fe954004eb0 con 0x7fe970146350 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9700afec0 con 0x7fe970146350 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe968001ec0 con 0x7fe970146350 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.725+0000 7fe962ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.722 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557594021 0 0) 0x7fe9701da6c0 con 0x7fe9700b4a00 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe954004eb0 con 0x7fe9700b4a00 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe96c001830 con 0x7fe9700b4a00 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.723 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 416493523 0 0) 0x7fe9701db2f0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9701da6c0 con 0x7fe9700a9ef0 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe964001970 con 0x7fe9700a9ef0 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.724 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3050268113 0 0) 0x7fe9700afec0 con 0x7fe970146350 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 >> v1:172.21.15.3:6790/0 conn(0x7fe9700a9ef0 legacy=0x7fe9701dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 >> v1:172.21.15.3:6789/0 conn(0x7fe9700b4a00 legacy=0x7fe9700ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 monclient: found mon.b 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700adf40 con 0x7fe970146350 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 10 monclient: _renew_subs 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9700aeca0 con 0x7fe970146350 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 10 monclient: _renew_subs 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9700b1400 con 0x7fe970146350 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe9784f6640 1 librados: init done 2024-03-29T11:14:36.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe968004e10 con 0x7fe970146350 2024-03-29T11:14:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fe968005120 con 0x7fe970146350 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 monclient: dump: 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.728 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe974a8f640 4 set_mon_vals no callback set 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe968005610 con 0x7fe970146350 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3540881140 0 0) 0x7fe968006a10 con 0x7fe970146350 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 10 client.4126.objecter ms_dispatch 0x7fe970000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.726+0000 7fe962ffd640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.727+0000 7fe9784f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.727+0000 7fe9784f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.727+0000 7fe9784f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.727+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe958005080 con 0x7fe970146350 2024-03-29T11:14:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.729+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe968006bd0 con 0x7fe970146350 2024-03-29T11:14:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.729+0000 7fe962ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.729+0000 7fe962ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.803+0000 7fe9784f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1cecc8b9-c752-4e32-9cd2-943602908be5", "id": 1}] 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.803+0000 7fe9784f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1cecc8b9-c752-4e32-9cd2-943602908be5", "id": 1}] 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.803+0000 7fe9784f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.803+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1cecc8b9-c752-4e32-9cd2-943602908be5", "id": 1} v 0) v1 -- 0x7fe9580058b0 con 0x7fe970146350 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.806+0000 7fe962ffd640 1 -- 172.21.15.32:0/3708652620 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1cecc8b9-c752-4e32-9cd2-943602908be5", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 4214125108 0 940483276) 0x7fe968005ae0 con 0x7fe970146350 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.806+0000 7fe962ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1cecc8b9-c752-4e32-9cd2-943602908be5", "id": 1}] 2024-03-29T11:14:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.806+0000 7fe962ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:36.787 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 10 librados: watch_flush enter 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 10 librados: watch_flush exit 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 10 monclient: shutdown 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 >> v1:172.21.15.32:6789/0 conn(0x7fe970146350 legacy=0x7fe9700b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 shutdown_connections 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 >> 172.21.15.32:0/3708652620 conn(0x7fe970066920 msgr2=0x7fe9700b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 shutdown_connections 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 -- 172.21.15.32:0/3708652620 wait complete. 2024-03-29T11:14:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.807+0000 7fe9784f6640 1 librados: shutdown 2024-03-29T11:14:36.795 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70 2 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.917+0000 7f689327b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.917+0000 7f689327b640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.917+0000 7f689327b640 10 monclient: monmap: 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:36.918962+0000 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:36.918962+0000 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.896 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.918+0000 7f689327b640 1 Processor -- start 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.918+0000 7f689327b640 1 -- start start 2024-03-29T11:14:36.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.918+0000 7f689327b640 10 monclient: init 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient(hunting): picked mon.noname-a con 0x7f688c0b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient(hunting): picked mon.noname-c con 0x7f688c0a9ef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient(hunting): picked mon.noname-b con 0x7f688c146350 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c145790 con 0x7f688c0b4a00 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c0aabe0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c0aaec0 con 0x7f688c146350 2024-03-29T11:14:36.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 10 monclient(hunting): authenticate will time out at 800.182191s 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6890816640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f688c0a9ef0 0x7f688c0b1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34760/0 (socket says 172.21.15.32:34760) 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6890816640 1 -- 172.21.15.32:0/2777391810 learned_addr learned my addr 172.21.15.32:0/2777391810 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619065221 0 0) 0x7f688c0aabe0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6878003520 con 0x7f688c0a9ef0 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083887332 0 0) 0x7f688c145790 con 0x7f688c0b4a00 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:14:36.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c0aabe0 con 0x7f688c0b4a00 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350500015 0 0) 0x7f688c0aaec0 con 0x7f688c146350 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c145790 con 0x7f688c146350 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2047340313 0 0) 0x7f6878003520 con 0x7f688c0a9ef0 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 >> v1:172.21.15.3:6790/0 conn(0x7f688c146350 legacy=0x7f688c14da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 >> v1:172.21.15.3:6789/0 conn(0x7f688c0b4a00 legacy=0x7f688c145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 monclient: found mon.noname-c 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c142170 con 0x7f688c0a9ef0 2024-03-29T11:14:36.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f689327b640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6888001690 con 0x7f688c0a9ef0 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.919+0000 7f6883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: dump: 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.909 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6888001870 con 0x7f688c0a9ef0 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 1 -- 172.21.15.32:0/2777391810 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6888001cc0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f6883fff640 10 monclient: dump: 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.910 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 4 set_mon_vals no callback set 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 10 monclient: shutdown 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 1 -- 172.21.15.32:0/2777391810 >> v1:172.21.15.32:6789/0 conn(0x7f688c0a9ef0 legacy=0x7f688c0b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 1 -- 172.21.15.32:0/2777391810 shutdown_connections 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 1 -- 172.21.15.32:0/2777391810 >> 172.21.15.32:0/2777391810 conn(0x7f688c066920 msgr2=0x7f688c066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 1 -- 172.21.15.32:0/2777391810 shutdown_connections 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.920+0000 7f689327b640 1 -- 172.21.15.32:0/2777391810 wait complete. 2024-03-29T11:14:36.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 10 monclient: build_initial_monmap 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 10 monclient: monmap: 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 librados: starting msgr at 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 librados: starting objecter 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 Processor -- start 2024-03-29T11:14:36.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 -- start start 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 librados: setting wanted keys 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 1 librados: calling monclient init 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.921+0000 7f689327b640 10 monclient: init 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient(hunting): picked mon.noname-b con 0x7f688c0a9ef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient(hunting): picked mon.noname-c con 0x7f688c0b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient(hunting): picked mon.noname-a con 0x7f688c146350 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c13d990 con 0x7f688c146350 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c0afab0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c0b0320 con 0x7f688c0b4a00 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f689327b640 10 monclient(hunting): authenticate will time out at 800.185344s 2024-03-29T11:14:36.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6891017640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f688c0a9ef0 0x7f688c0b09d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34762/0 (socket says 172.21.15.32:34762) 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6891017640 1 -- 172.21.15.32:0/311896166 learned_addr learned my addr 172.21.15.32:0/311896166 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter ms_handle_connect 0x7f688c0a9ef0 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter ms_handle_connect 0x7f688c0b4a00 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390783345 0 0) 0x7f688c0afab0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6870004eb0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter ms_handle_connect 0x7f688c146350 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010336259 0 0) 0x7f688c0b0320 con 0x7f688c0b4a00 2024-03-29T11:14:36.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c0afab0 con 0x7f688c0b4a00 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218181518 0 0) 0x7f688c13d990 con 0x7f688c146350 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:36.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c0b0320 con 0x7f688c146350 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433494258 0 0) 0x7f6870004eb0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f688c13d990 con 0x7f688c0a9ef0 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f687c001310 con 0x7f688c0a9ef0 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.916 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.922+0000 7f6881ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3861015248 0 0) 0x7f688c0afab0 con 0x7f688c0b4a00 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6870004eb0 con 0x7f688c0b4a00 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6888001b80 con 0x7f688c0b4a00 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.917 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1230186132 0 0) 0x7f688c0b0320 con 0x7f688c146350 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f688c0afab0 con 0x7f688c146350 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6884001bd0 con 0x7f688c146350 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.918 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3152134940 0 0) 0x7f688c13d990 con 0x7f688c0a9ef0 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 >> v1:172.21.15.3:6790/0 conn(0x7f688c0b4a00 legacy=0x7f688c13cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 >> v1:172.21.15.3:6789/0 conn(0x7f688c146350 legacy=0x7f688c1da510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 monclient: found mon.b 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:36.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0b04b0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 10 monclient: _renew_subs 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c13d1f0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f687c0035f0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f687c0039c0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 monclient: dump: 2024-03-29T11:14:36.920 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f687c003f30 con 0x7f688c0a9ef0 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 10 monclient: _renew_subs 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f68837fe640 4 set_mon_vals no callback set 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c1dac20 con 0x7f688c0a9ef0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f689327b640 1 librados: init done 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 40692095 0 0) 0x7f687c005ba0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 10 client.4132.objecter ms_dispatch 0x7f688c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.923+0000 7f6881ffb640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.924+0000 7f689327b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.924+0000 7f689327b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.924+0000 7f689327b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.924+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c13d7f0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.926+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f688c13d7f0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.926+0000 7f6881ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:36.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.926+0000 7f6881ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:36.951 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:36.973+0000 7f2eac012040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:14:36.952 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:36.975+0000 7f5c76038040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:14:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.999+0000 7f689327b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70", "id": 2}] 2024-03-29T11:14:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.999+0000 7f689327b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70", "id": 2}] 2024-03-29T11:14:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.999+0000 7f689327b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:36.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:36.999+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70", "id": 2} v 0) v1 -- 0x7f688c06a3f0 con 0x7f688c0a9ef0 2024-03-29T11:14:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.002+0000 7f6881ffb640 1 -- 172.21.15.32:0/311896166 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 577015343 0 216599125) 0x7f687c005780 con 0x7f688c0a9ef0 2024-03-29T11:14:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.002+0000 7f6881ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70", "id": 2}] 2024-03-29T11:14:36.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.002+0000 7f6881ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:36.980 INFO:teuthology.orchestra.run.smithi032.stdout:2 2024-03-29T11:14:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 10 librados: watch_flush enter 2024-03-29T11:14:36.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 10 librados: watch_flush exit 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 10 monclient: shutdown 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 >> v1:172.21.15.32:6789/0 conn(0x7f688c0a9ef0 legacy=0x7f688c0b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 shutdown_connections 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 >> 172.21.15.32:0/311896166 conn(0x7f688c066920 msgr2=0x7f688c066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 shutdown_connections 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 -- 172.21.15.32:0/311896166 wait complete. 2024-03-29T11:14:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.003+0000 7f689327b640 1 librados: shutdown 2024-03-29T11:14:36.991 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new 2a3ca543-5fae-4475-bd72-a4bb45c9d0aa 3 2024-03-29T11:14:37.009 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.032+0000 7f5c76038040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:14:37.009 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.031+0000 7f2eac012040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:14:37.055 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.078+0000 7f5c76038040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:14:37.058 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.080+0000 7f2eac012040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.114+0000 7fa81a54d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.114+0000 7fa81a54d640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.114+0000 7fa81a54d640 10 monclient: monmap: 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.092 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:37.115314+0000 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:37.115314+0000 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 1 Processor -- start 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 1 -- start start 2024-03-29T11:14:37.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient: init 2024-03-29T11:14:37.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-a con 0x7fa8140b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8140a9ef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-b con 0x7fa814146350 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa814145790 con 0x7fa8140b4a00 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8140aabe0 con 0x7fa8140a9ef0 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8140aaec0 con 0x7fa814146350 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa81a54d640 10 monclient(hunting): authenticate will time out at 800.378635s 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa8137fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8140a9ef0 0x7fa8140b1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34766/0 (socket says 172.21.15.32:34766) 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.115+0000 7fa8137fe640 1 -- 172.21.15.32:0/877214929 learned_addr learned my addr 172.21.15.32:0/877214929 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503141912 0 0) 0x7fa8140aabe0 con 0x7fa8140a9ef0 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800003520 con 0x7fa8140a9ef0 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175782488 0 0) 0x7fa8140aaec0 con 0x7fa814146350 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8140aabe0 con 0x7fa814146350 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973592001 0 0) 0x7fa814145790 con 0x7fa8140b4a00 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:14:37.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8140aaec0 con 0x7fa8140b4a00 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2762758904 0 0) 0x7fa800003520 con 0x7fa8140a9ef0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 >> v1:172.21.15.3:6790/0 conn(0x7fa814146350 legacy=0x7fa81414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 >> v1:172.21.15.3:6789/0 conn(0x7fa8140b4a00 legacy=0x7fa814145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 monclient: found mon.noname-c 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa814142170 con 0x7fa8140a9ef0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa7fc001690 con 0x7fa8140a9ef0 2024-03-29T11:14:37.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: dump: 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7fc001870 con 0x7fa8140a9ef0 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 1 -- 172.21.15.32:0/877214929 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa7fc001c00 con 0x7fa8140a9ef0 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 4 set_mon_vals no callback set 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa812ffd640 10 monclient: dump: 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.109 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 10 monclient: shutdown 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.116+0000 7fa81a54d640 1 -- 172.21.15.32:0/877214929 >> v1:172.21.15.32:6789/0 conn(0x7fa8140a9ef0 legacy=0x7fa8140b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 -- 172.21.15.32:0/877214929 shutdown_connections 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 -- 172.21.15.32:0/877214929 >> 172.21.15.32:0/877214929 conn(0x7fa814066920 msgr2=0x7fa814066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 -- 172.21.15.32:0/877214929 shutdown_connections 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 -- 172.21.15.32:0/877214929 wait complete. 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 10 monclient: monmap: 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.110 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 librados: starting msgr at 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.117+0000 7fa81a54d640 1 librados: starting objecter 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 Processor -- start 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 -- start start 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 librados: setting wanted keys 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 librados: calling monclient init 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient: init 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-c con 0x7fa8140a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-b con 0x7fa8140b4a00 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient(hunting): picked mon.noname-a con 0x7fa814146350 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8141da600 con 0x7fa814146350 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8140ad5b0 con 0x7fa8140b4a00 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8141db230 con 0x7fa8140a9ef0 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa81a54d640 10 monclient(hunting): authenticate will time out at 800.381688s 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa8137fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa8140b4a00 0x7fa81413ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34768/0 (socket says 172.21.15.32:34768) 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa8137fe640 1 -- 172.21.15.32:0/3184425523 learned_addr learned my addr 172.21.15.32:0/3184425523 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa810ff9640 10 client.?.objecter ms_handle_connect 0x7fa8140b4a00 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa810ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa810ff9640 10 client.?.objecter ms_handle_connect 0x7fa8140a9ef0 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.118+0000 7fa810ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 client.?.objecter ms_handle_connect 0x7fa814146350 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876142285 0 0) 0x7fa8140ad5b0 con 0x7fa8140b4a00 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7f4004eb0 con 0x7fa8140b4a00 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294514449 0 0) 0x7fa8141db230 con 0x7fa8140a9ef0 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8140ad5b0 con 0x7fa8140a9ef0 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172107033 0 0) 0x7fa8141da600 con 0x7fa814146350 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8141db230 con 0x7fa814146350 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428538851 0 0) 0x7fa7f4004eb0 con 0x7fa8140b4a00 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8141da600 con 0x7fa8140b4a00 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa7fc0015d0 con 0x7fa8140b4a00 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.114 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3024057814 0 0) 0x7fa8140ad5b0 con 0x7fa8140a9ef0 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7f4004eb0 con 0x7fa8140a9ef0 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa804001870 con 0x7fa8140a9ef0 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.115 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2538259948 0 0) 0x7fa8141db230 con 0x7fa814146350 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8140ad5b0 con 0x7fa814146350 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa80c001a40 con 0x7fa814146350 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.116 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3087503745 0 0) 0x7fa8141da600 con 0x7fa8140b4a00 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 >> v1:172.21.15.3:6790/0 conn(0x7fa8140a9ef0 legacy=0x7fa8141dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 >> v1:172.21.15.3:6789/0 conn(0x7fa814146350 legacy=0x7fa8140ae020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 monclient: found mon.b 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8141db3c0 con 0x7fa8140b4a00 2024-03-29T11:14:37.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 10 monclient: _renew_subs 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8140b00a0 con 0x7fa8140b4a00 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 10 monclient: _renew_subs 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81413b740 con 0x7fa8140b4a00 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa81a54d640 1 librados: init done 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7fc0011f0 con 0x7fa8140b4a00 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa7fc004c30 con 0x7fa8140b4a00 2024-03-29T11:14:37.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa811ffb640 4 set_mon_vals no callback set 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 monclient: dump: 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa7fc0051a0 con 0x7fa8140b4a00 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1501411633 0 0) 0x7fa7fc006860 con 0x7fa8140b4a00 2024-03-29T11:14:37.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 10 client.4138.objecter ms_dispatch 0x7fa814000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.119+0000 7fa810ff9640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.120+0000 7fa81a54d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.120+0000 7fa81a54d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.120+0000 7fa81a54d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.120+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8005080 con 0x7fa8140b4a00 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.122+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa7fc006480 con 0x7fa8140b4a00 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.122+0000 7fa810ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.122+0000 7fa810ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:37.121 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.138+0000 7f2eac012040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:14:37.121 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.134+0000 7f5c76038040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:14:37.155 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.178+0000 7f5c76038040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:14:37.160 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.183+0000 7f2eac012040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:14:37.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.195+0000 7fa81a54d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2a3ca543-5fae-4475-bd72-a4bb45c9d0aa", "id": 3}] 2024-03-29T11:14:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.195+0000 7fa81a54d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2a3ca543-5fae-4475-bd72-a4bb45c9d0aa", "id": 3}] 2024-03-29T11:14:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.195+0000 7fa81a54d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.195+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "2a3ca543-5fae-4475-bd72-a4bb45c9d0aa", "id": 3} v 0) v1 -- 0x7fa7f80058b0 con 0x7fa8140b4a00 2024-03-29T11:14:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.198+0000 7fa810ff9640 1 -- 172.21.15.32:0/3184425523 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a3ca543-5fae-4475-bd72-a4bb45c9d0aa", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2377336126 0 525046306) 0x7fa7fc006a70 con 0x7fa8140b4a00 2024-03-29T11:14:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.198+0000 7fa810ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2a3ca543-5fae-4475-bd72-a4bb45c9d0aa", "id": 3}] 2024-03-29T11:14:37.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.198+0000 7fa810ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:37.176 INFO:teuthology.orchestra.run.smithi032.stdout:3 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 10 librados: watch_flush enter 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 10 librados: watch_flush exit 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 10 monclient: shutdown 2024-03-29T11:14:37.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 >> v1:172.21.15.32:6789/0 conn(0x7fa8140b4a00 legacy=0x7fa81413ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 shutdown_connections 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 >> 172.21.15.32:0/3184425523 conn(0x7fa814066920 msgr2=0x7fa8140b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 shutdown_connections 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 -- 172.21.15.32:0/3184425523 wait complete. 2024-03-29T11:14:37.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.199+0000 7fa81a54d640 1 librados: shutdown 2024-03-29T11:14:37.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new e3918c69-a425-45af-aa06-652fea1f9efa 4 2024-03-29T11:14:37.275 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.297+0000 7f5c76038040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:14:37.284 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.306+0000 7f2eac012040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:14:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.350+0000 7f79489cb640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.350+0000 7f79489cb640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.350+0000 7f79489cb640 10 monclient: monmap: 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:37.351541+0000 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:37.351541+0000 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 1 Processor -- start 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 1 -- start start 2024-03-29T11:14:37.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient: init 2024-03-29T11:14:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-a con 0x7f79440b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-b con 0x7f79440a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-c con 0x7f7944146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7944145790 con 0x7f79440b4a00 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440aabe0 con 0x7f7944146350 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440aaec0 con 0x7f79440a9ef0 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f79489cb640 10 monclient(hunting): authenticate will time out at 800.614753s 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f7942d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7944146350 0x7f794414da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34774/0 (socket says 172.21.15.32:34774) 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.351+0000 7f7942d9d640 1 -- 172.21.15.32:0/1617232204 learned_addr learned my addr 172.21.15.32:0/1617232204 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716341731 0 0) 0x7f79440aabe0 con 0x7f7944146350 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:14:37.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f792c003520 con 0x7f7944146350 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119944978 0 0) 0x7f7944145790 con 0x7f79440b4a00 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79440aabe0 con 0x7f79440b4a00 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927163875 0 0) 0x7f79440aaec0 con 0x7f79440a9ef0 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:14:37.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7944145790 con 0x7f79440a9ef0 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3618536834 0 0) 0x7f79440aabe0 con 0x7f79440b4a00 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 >> v1:172.21.15.3:6790/0 conn(0x7f79440a9ef0 legacy=0x7f79440b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 >> v1:172.21.15.32:6789/0 conn(0x7f7944146350 legacy=0x7f794414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 monclient: found mon.noname-a 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944142170 con 0x7f79440b4a00 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f79489cb640 5 monclient: authenticate success, global_id 4137 2024-03-29T11:14:37.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7934001670 con 0x7f79440b4a00 2024-03-29T11:14:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f79489cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: dump: 2024-03-29T11:14:37.344 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f79489cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79340017d0 con 0x7f79440b4a00 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f79489cb640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 1 -- 172.21.15.32:0/1617232204 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7934001c20 con 0x7f79440b4a00 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f79489cb640 4 set_mon_vals no callback set 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.352+0000 7f794159a640 10 monclient: dump: 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.346 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 10 monclient: shutdown 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 1 -- 172.21.15.32:0/1617232204 >> v1:172.21.15.3:6789/0 conn(0x7f79440b4a00 legacy=0x7f7944145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 1 -- 172.21.15.32:0/1617232204 shutdown_connections 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 1 -- 172.21.15.32:0/1617232204 >> 172.21.15.32:0/1617232204 conn(0x7f7944066920 msgr2=0x7f7944066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 1 -- 172.21.15.32:0/1617232204 shutdown_connections 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.353+0000 7f79489cb640 1 -- 172.21.15.32:0/1617232204 wait complete. 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 10 monclient: monmap: 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 librados: starting msgr at 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 librados: starting objecter 2024-03-29T11:14:37.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 Processor -- start 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 -- start start 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 librados: setting wanted keys 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 1 librados: calling monclient init 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.354+0000 7f79489cb640 10 monclient: init 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-c con 0x7f79440a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-b con 0x7f79440b4a00 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient(hunting): picked mon.noname-a con 0x7f7944146350 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440ae860 con 0x7f7944146350 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794413e8e0 con 0x7f79440b4a00 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79440af880 con 0x7f79440a9ef0 2024-03-29T11:14:37.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f79489cb640 10 monclient(hunting): authenticate will time out at 800.618075s 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f794259c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f79440a9ef0 0x7f79440ad1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.32:35262/0 (socket says 172.21.15.32:35262) 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f794259c640 1 -- 172.21.15.32:0/501488703 learned_addr learned my addr 172.21.15.32:0/501488703 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter ms_handle_connect 0x7f79440a9ef0 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter ms_handle_connect 0x7f79440b4a00 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter ms_handle_connect 0x7f7944146350 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595594724 0 0) 0x7f79440af880 con 0x7f79440a9ef0 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7924004eb0 con 0x7f79440a9ef0 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064934960 0 0) 0x7f79440ae860 con 0x7f7944146350 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79440af880 con 0x7f7944146350 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806601558 0 0) 0x7f794413e8e0 con 0x7f79440b4a00 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79440ae860 con 0x7f79440b4a00 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080724446 0 0) 0x7f7924004eb0 con 0x7f79440a9ef0 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f794413e8e0 con 0x7f79440a9ef0 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7934001670 con 0x7f79440a9ef0 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.352 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 784306290 0 0) 0x7f79440af880 con 0x7f7944146350 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7924004eb0 con 0x7f7944146350 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f793c001ab0 con 0x7f7944146350 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 620050797 0 0) 0x7f79440ae860 con 0x7f79440b4a00 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79440af880 con 0x7f79440b4a00 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7938001a10 con 0x7f79440b4a00 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.355 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1157713528 0 0) 0x7f794413e8e0 con 0x7f79440a9ef0 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 >> v1:172.21.15.32:6789/0 conn(0x7f79440b4a00 legacy=0x7f794413dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.355+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 >> v1:172.21.15.3:6789/0 conn(0x7f7944146350 legacy=0x7f794413e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 monclient: found mon.c 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79440afca0 con 0x7f79440a9ef0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 5 monclient: authenticate success, global_id 4136 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 monclient: _renew_subs 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79440addf0 con 0x7f79440a9ef0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 client.4136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 monclient: _renew_subs 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79440adaa0 con 0x7f79440a9ef0 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 1 librados: init done 2024-03-29T11:14:37.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79340039e0 con 0x7f79440a9ef0 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7934003e30 con 0x7f79440a9ef0 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 monclient: dump: 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.358 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7940d99640 4 set_mon_vals no callback set 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7934005780 con 0x7f79440a9ef0 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4287489630 0 0) 0x7f79340069c0 con 0x7f79440a9ef0 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 10 client.4136.objecter ms_dispatch 0x7f7944000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:14:37.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 3 client.4136.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 3 client.4136.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f7932ffd640 20 client.4136.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.356+0000 7f79489cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.357+0000 7f79489cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.357+0000 7f79489cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.357+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79440ae190 con 0x7f79440a9ef0 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.359+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f79340064b0 con 0x7f79440a9ef0 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.359+0000 7f7932ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.359+0000 7f7932ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:37.361 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.382+0000 7f2eac012040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:14:37.361 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.370+0000 7f5c76038040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:14:37.395 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.418+0000 7f5c76038040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:14:37.408 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.430+0000 7f2eac012040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:14:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.432+0000 7f79489cb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e3918c69-a425-45af-aa06-652fea1f9efa", "id": 4}] 2024-03-29T11:14:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.432+0000 7f79489cb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e3918c69-a425-45af-aa06-652fea1f9efa", "id": 4}] 2024-03-29T11:14:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.432+0000 7f79489cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.432+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd new", "uuid": "e3918c69-a425-45af-aa06-652fea1f9efa", "id": 4} v 0) v1 -- 0x7f794406a3f0 con 0x7f79440a9ef0 2024-03-29T11:14:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.435+0000 7f7932ffd640 1 -- 172.21.15.32:0/501488703 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e3918c69-a425-45af-aa06-652fea1f9efa", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 92236238 0 1697012583) 0x7f79340066b0 con 0x7f79440a9ef0 2024-03-29T11:14:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.435+0000 7f7932ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e3918c69-a425-45af-aa06-652fea1f9efa", "id": 4}] 2024-03-29T11:14:37.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.435+0000 7f7932ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:37.413 INFO:teuthology.orchestra.run.smithi032.stdout:4 2024-03-29T11:14:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 10 librados: watch_flush enter 2024-03-29T11:14:37.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 10 librados: watch_flush exit 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 20 client.4136.objecter shutdown clearing up homeless session... 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 10 client.4136.objecter successfully canceled tick 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 10 monclient: shutdown 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 >> v1:172.21.15.3:6790/0 conn(0x7f79440a9ef0 legacy=0x7f79440ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 shutdown_connections 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 >> 172.21.15.32:0/501488703 conn(0x7f7944066920 msgr2=0x7f79440b8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 shutdown_connections 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 -- 172.21.15.32:0/501488703 wait complete. 2024-03-29T11:14:37.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.436+0000 7f79489cb640 1 librados: shutdown 2024-03-29T11:14:37.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new dcc5177c-2cec-4390-899d-ddee6450a1e8 5 2024-03-29T11:14:37.450 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.473+0000 7f5c76038040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:14:37.464 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.487+0000 7f2eac012040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:14:37.499 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.521+0000 7f5c76038040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:14:37.515 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.537+0000 7f2eac012040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:14:37.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.546+0000 7ff0c878d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.546+0000 7ff0c878d640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.547+0000 7ff0c878d640 10 monclient: monmap: 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:37.548045+0000 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:37.548045+0000 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.525 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.547+0000 7ff0c878d640 1 Processor -- start 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.547+0000 7ff0c878d640 1 -- start start 2024-03-29T11:14:37.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.547+0000 7ff0c878d640 10 monclient: init 2024-03-29T11:14:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:37.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c00b4a00 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c00a9ef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c0146350 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c0145790 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00aabe0 con 0x7ff0c00b4a00 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00aaec0 con 0x7ff0c0146350 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c878d640 10 monclient(hunting): authenticate will time out at 800.811346s 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c6529640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0c00b4a00 0x7ff0c0145c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34788/0 (socket says 172.21.15.32:34788) 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c6529640 1 -- 172.21.15.32:0/1911485761 learned_addr learned my addr 172.21.15.32:0/1911485761 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526042923 0 0) 0x7ff0c00aabe0 con 0x7ff0c00b4a00 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0ac003520 con 0x7ff0c00b4a00 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22910834 0 0) 0x7ff0c00aaec0 con 0x7ff0c0146350 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c00aabe0 con 0x7ff0c0146350 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431942465 0 0) 0x7ff0c0145790 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.548+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c00aaec0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378545248 0 0) 0x7ff0ac003520 con 0x7ff0c00b4a00 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 >> v1:172.21.15.3:6790/0 conn(0x7ff0c0146350 legacy=0x7ff0c014da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 >> v1:172.21.15.3:6789/0 conn(0x7ff0c00a9ef0 legacy=0x7ff0c00b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 monclient: found mon.noname-c 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c0142170 con 0x7ff0c00b4a00 2024-03-29T11:14:37.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0bc001690 con 0x7ff0c00b4a00 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: dump: 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.544 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0bc001870 con 0x7ff0c00b4a00 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 1 -- 172.21.15.32:0/1911485761 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0bc001c80 con 0x7ff0c00b4a00 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 4 set_mon_vals no callback set 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c5527640 10 monclient: dump: 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.546 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 10 monclient: shutdown 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 1 -- 172.21.15.32:0/1911485761 >> v1:172.21.15.32:6789/0 conn(0x7ff0c00b4a00 legacy=0x7ff0c0145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 1 -- 172.21.15.32:0/1911485761 shutdown_connections 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 1 -- 172.21.15.32:0/1911485761 >> 172.21.15.32:0/1911485761 conn(0x7ff0c0066920 msgr2=0x7ff0c0066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 1 -- 172.21.15.32:0/1911485761 shutdown_connections 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.549+0000 7ff0c878d640 1 -- 172.21.15.32:0/1911485761 wait complete. 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 10 monclient: monmap: 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:37.547 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 librados: starting msgr at 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 librados: starting objecter 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 Processor -- start 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 -- start start 2024-03-29T11:14:37.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 librados: setting wanted keys 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 1 librados: calling monclient init 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.550+0000 7ff0c878d640 10 monclient: init 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c00a9ef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c00b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c0146350 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00afd80 con 0x7ff0c0146350 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00af010 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c00affb0 con 0x7ff0c00b4a00 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c878d640 10 monclient(hunting): authenticate will time out at 800.814242s 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c6529640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0c00a9ef0 0x7ff0c013ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34794/0 (socket says 172.21.15.32:34794) 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0c6529640 1 -- 172.21.15.32:0/1679174652 learned_addr learned my addr 172.21.15.32:0/1679174652 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter ms_handle_connect 0x7ff0c00a9ef0 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter ms_handle_connect 0x7ff0c0146350 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter ms_handle_connect 0x7ff0c00b4a00 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602070851 0 0) 0x7ff0c00af010 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:14:37.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a4004eb0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407230946 0 0) 0x7ff0c00affb0 con 0x7ff0c00b4a00 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c00af010 con 0x7ff0c00b4a00 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959539724 0 0) 0x7ff0c00afd80 con 0x7ff0c0146350 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c00affb0 con 0x7ff0c0146350 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 615698109 0 0) 0x7ff0a4004eb0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c00afd80 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0bc001bc0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.552 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1538004050 0 0) 0x7ff0c00af010 con 0x7ff0c00b4a00 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0a4004eb0 con 0x7ff0c00b4a00 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b4001870 con 0x7ff0c00b4a00 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.551+0000 7ff0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.554 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548185187 0 0) 0x7ff0c00affb0 con 0x7ff0c0146350 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0c00af010 con 0x7ff0c0146350 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b80019c0 con 0x7ff0c0146350 2024-03-29T11:14:37.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.556 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1089620345 0 0) 0x7ff0c00afd80 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 >> v1:172.21.15.3:6790/0 conn(0x7ff0c00b4a00 legacy=0x7ff0c01d7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 >> v1:172.21.15.3:6789/0 conn(0x7ff0c0146350 legacy=0x7ff0c01e2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 monclient: found mon.b 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c00b06b0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 10 monclient: _renew_subs 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c00ae5d0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 10 monclient: _renew_subs 2024-03-29T11:14:37.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c013b560 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c878d640 1 librados: init done 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0bc004bf0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0bc004f00 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 monclient: dump: 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.560 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff0bc0053f0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0c4d26640 4 set_mon_vals no callback set 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 676881986 0 0) 0x7ff0bc0069c0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 10 client.4150.objecter ms_dispatch 0x7ff0c0000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.552+0000 7ff0b2ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.553+0000 7ff0c878d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.553+0000 7ff0c878d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.553+0000 7ff0c878d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.553+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a8005080 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.555+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff0bc006bd0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.555+0000 7ff0b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.555+0000 7ff0b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.628+0000 7ff0c878d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dcc5177c-2cec-4390-899d-ddee6450a1e8", "id": 5}] 2024-03-29T11:14:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.628+0000 7ff0c878d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dcc5177c-2cec-4390-899d-ddee6450a1e8", "id": 5}] 2024-03-29T11:14:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.628+0000 7ff0c878d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.628+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dcc5177c-2cec-4390-899d-ddee6450a1e8", "id": 5} v 0) v1 -- 0x7ff0a80058b0 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.631+0000 7ff0b2ffd640 1 -- 172.21.15.32:0/1679174652 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dcc5177c-2cec-4390-899d-ddee6450a1e8", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2794309742 0 1988412176) 0x7ff0bc006510 con 0x7ff0c00a9ef0 2024-03-29T11:14:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.631+0000 7ff0b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dcc5177c-2cec-4390-899d-ddee6450a1e8", "id": 5}] 2024-03-29T11:14:37.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.631+0000 7ff0b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:37.609 INFO:teuthology.orchestra.run.smithi032.stdout:5 2024-03-29T11:14:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 10 librados: watch_flush enter 2024-03-29T11:14:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 10 librados: watch_flush exit 2024-03-29T11:14:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:14:37.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 10 monclient: shutdown 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 >> v1:172.21.15.32:6789/0 conn(0x7ff0c00a9ef0 legacy=0x7ff0c013ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 shutdown_connections 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 >> 172.21.15.32:0/1679174652 conn(0x7ff0c0066920 msgr2=0x7ff0c00b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 shutdown_connections 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 -- 172.21.15.32:0/1679174652 wait complete. 2024-03-29T11:14:37.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.632+0000 7ff0c878d640 1 librados: shutdown 2024-03-29T11:14:37.620 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new d2642940-19a3-43e0-8402-e924f9fbe3b4 6 2024-03-29T11:14:37.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.783+0000 7f41bcfaf640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.783+0000 7f41bcfaf640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.783+0000 7f41bcfaf640 10 monclient: monmap: 2024-03-29T11:14:37.761 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.761 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:37.784572+0000 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:37.784572+0000 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:37.762 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 1 Processor -- start 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 1 -- start start 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient: init 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f41b80b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f41b80a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f41b8146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b8145790 con 0x7f41b80b4a00 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80aabe0 con 0x7f41b8146350 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80aaec0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41bcfaf640 10 monclient(hunting): authenticate will time out at 801.047742s 2024-03-29T11:14:37.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41b6d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f41b8146350 0x7f41b814da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34798/0 (socket says 172.21.15.32:34798) 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.784+0000 7f41b6d9d640 1 -- 172.21.15.32:0/1278188118 learned_addr learned my addr 172.21.15.32:0/1278188118 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234900040 0 0) 0x7f41b80aabe0 con 0x7f41b8146350 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41a4003520 con 0x7f41b8146350 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581904770 0 0) 0x7f41b80aaec0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b80aabe0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198061948 0 0) 0x7f41b8145790 con 0x7f41b80b4a00 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b80aaec0 con 0x7f41b80b4a00 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178142590 0 0) 0x7f41a4003520 con 0x7f41b8146350 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 >> v1:172.21.15.3:6790/0 conn(0x7f41b80a9ef0 legacy=0x7f41b80b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 >> v1:172.21.15.3:6789/0 conn(0x7f41b80b4a00 legacy=0x7f41b8145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b8142170 con 0x7f41b8146350 2024-03-29T11:14:37.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41b0001a10 con 0x7f41b8146350 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41bcfaf640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41bcfaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: dump: 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.775 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41bcfaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41b0001bf0 con 0x7f41b8146350 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41bcfaf640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 1 -- 172.21.15.32:0/1278188118 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41b0002000 con 0x7f41b8146350 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41bcfaf640 4 set_mon_vals no callback set 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.785+0000 7f41b5d9b640 10 monclient: dump: 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.776 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 10 monclient: shutdown 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 1 -- 172.21.15.32:0/1278188118 >> v1:172.21.15.32:6789/0 conn(0x7f41b8146350 legacy=0x7f41b814da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 1 -- 172.21.15.32:0/1278188118 shutdown_connections 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 1 -- 172.21.15.32:0/1278188118 >> 172.21.15.32:0/1278188118 conn(0x7f41b8066920 msgr2=0x7f41b8066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 1 -- 172.21.15.32:0/1278188118 shutdown_connections 2024-03-29T11:14:37.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 1 -- 172.21.15.32:0/1278188118 wait complete. 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.786+0000 7f41bcfaf640 10 monclient: monmap: 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 librados: starting msgr at 2024-03-29T11:14:37.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 librados: starting objecter 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 Processor -- start 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 -- start start 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 librados: setting wanted keys 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 librados: calling monclient init 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient: init 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-c con 0x7f41b80a9ef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-a con 0x7f41b80b4a00 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient(hunting): picked mon.noname-b con 0x7f41b8146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80afbc0 con 0x7f41b80b4a00 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80ae010 con 0x7f41b8146350 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41b80b0bc0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.787+0000 7f41bcfaf640 10 monclient(hunting): authenticate will time out at 801.050819s 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41b6d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f41b8146350 0x7f41b81d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34812/0 (socket says 172.21.15.32:34812) 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41b6d9d640 1 -- 172.21.15.32:0/2841800206 learned_addr learned my addr 172.21.15.32:0/2841800206 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter ms_handle_connect 0x7f41b8146350 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter ms_handle_connect 0x7f41b80a9ef0 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter ms_handle_connect 0x7f41b80b4a00 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366213918 0 0) 0x7f41b80b0bc0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4198004eb0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238210530 0 0) 0x7f41b80ae010 con 0x7f41b8146350 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b80b0bc0 con 0x7f41b8146350 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878959396 0 0) 0x7f41b80afbc0 con 0x7f41b80b4a00 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41b80ae010 con 0x7f41b80b4a00 2024-03-29T11:14:37.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185501479 0 0) 0x7f4198004eb0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b80afbc0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41a8001870 con 0x7f41b80a9ef0 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.782 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1501192515 0 0) 0x7f41b80b0bc0 con 0x7f41b8146350 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4198004eb0 con 0x7f41b8146350 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41b0001f00 con 0x7f41b8146350 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.783 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.784 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1533838333 0 0) 0x7f41b80ae010 con 0x7f41b80b4a00 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41b80b0bc0 con 0x7f41b80b4a00 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41a0001870 con 0x7f41b80b4a00 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.785 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1688622109 0 0) 0x7f41b80afbc0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 >> v1:172.21.15.32:6789/0 conn(0x7f41b8146350 legacy=0x7f41b81d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 >> v1:172.21.15.3:6789/0 conn(0x7f41b80b4a00 legacy=0x7f41b81d7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 monclient: found mon.c 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b80af0b0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 5 monclient: authenticate success, global_id 4148 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 10 monclient: _renew_subs 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b813c9b0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 10 client.4148.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 10 monclient: _renew_subs 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b80aec90 con 0x7f41b80a9ef0 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41bcfaf640 1 librados: init done 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41a8003610 con 0x7f41b80a9ef0 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f41a8003920 con 0x7f41b80a9ef0 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41b559a640 4 set_mon_vals no callback set 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 monclient: dump: 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.789 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41a8003e90 con 0x7f41b80a9ef0 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 73376662 0 0) 0x7f41a80056c0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 10 client.4148.objecter ms_dispatch 0x7f41b8000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 3 client.4148.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.788+0000 7f41aeffd640 3 client.4148.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.789+0000 7f41aeffd640 20 client.4148.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.789+0000 7f41bcfaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.789+0000 7f41bcfaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.789+0000 7f41bcfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.789+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c005080 con 0x7f41b80a9ef0 2024-03-29T11:14:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.791+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41a80051e0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.791+0000 7f41aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.791+0000 7f41aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.867+0000 7f41bcfaf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d2642940-19a3-43e0-8402-e924f9fbe3b4", "id": 6}] 2024-03-29T11:14:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.867+0000 7f41bcfaf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d2642940-19a3-43e0-8402-e924f9fbe3b4", "id": 6}] 2024-03-29T11:14:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.867+0000 7f41bcfaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:37.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.867+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd new", "uuid": "d2642940-19a3-43e0-8402-e924f9fbe3b4", "id": 6} v 0) v1 -- 0x7f419c0058b0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.869+0000 7f41aeffd640 1 -- 172.21.15.32:0/2841800206 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d2642940-19a3-43e0-8402-e924f9fbe3b4", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1590090892 0 1113811849) 0x7f41a80054d0 con 0x7f41b80a9ef0 2024-03-29T11:14:37.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.869+0000 7f41aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d2642940-19a3-43e0-8402-e924f9fbe3b4", "id": 6}] 2024-03-29T11:14:37.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.869+0000 7f41aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:37.848 INFO:teuthology.orchestra.run.smithi032.stdout:6 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 10 librados: watch_flush enter 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 10 librados: watch_flush exit 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 20 client.4148.objecter shutdown clearing up homeless session... 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 10 client.4148.objecter successfully canceled tick 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 10 monclient: shutdown 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 >> v1:172.21.15.3:6790/0 conn(0x7f41b80a9ef0 legacy=0x7f41b813bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 shutdown_connections 2024-03-29T11:14:37.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 >> 172.21.15.32:0/2841800206 conn(0x7f41b8066920 msgr2=0x7f41b813f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 shutdown_connections 2024-03-29T11:14:37.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 -- 172.21.15.32:0/2841800206 wait complete. 2024-03-29T11:14:37.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.870+0000 7f41bcfaf640 1 librados: shutdown 2024-03-29T11:14:37.858 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd new 9a2d4629-bd65-420b-855e-3ca59aa5c06f 7 2024-03-29T11:14:37.895 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:37.917+0000 7f5c76038040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:14:37.924 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:37.947+0000 7f2eac012040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:14:37.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 10 monclient: monmap: 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:14:37.983249+0000 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:14:37.983249+0000 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:37.960 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 1 Processor -- start 2024-03-29T11:14:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 1 -- start start 2024-03-29T11:14:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.982+0000 7f69bb77b640 10 monclient: init 2024-03-29T11:14:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-b con 0x7f69b4071f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-c con 0x7f69b40aeaf0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-a con 0x7f69b40af520 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b40b0d50 con 0x7f69b40af520 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b413fb80 con 0x7f69b40aeaf0 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b413fd10 con 0x7f69b4071f20 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69bb77b640 10 monclient(hunting): authenticate will time out at 801.246474s 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69b8d16640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f69b40aeaf0 0x7f69b40b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34828/0 (socket says 172.21.15.32:34828) 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69b8d16640 1 -- 172.21.15.32:0/788228843 learned_addr learned my addr 172.21.15.32:0/788228843 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171590142 0 0) 0x7f69b413fb80 con 0x7f69b40aeaf0 2024-03-29T11:14:37.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 10 monclient(hunting): my global_id is 4159 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.983+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69a0003520 con 0x7f69b40aeaf0 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480909323 0 0) 0x7f69b413fd10 con 0x7f69b4071f20 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b413fb80 con 0x7f69b4071f20 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716138322 0 0) 0x7f69b40b0d50 con 0x7f69b40af520 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b413fd10 con 0x7f69b40af520 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428119976 0 0) 0x7f69a0003520 con 0x7f69b40aeaf0 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 >> v1:172.21.15.3:6790/0 conn(0x7f69b4071f20 legacy=0x7f69b40b1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 >> v1:172.21.15.3:6789/0 conn(0x7f69b40af520 legacy=0x7f69b413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 monclient: found mon.noname-c 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b4151b30 con 0x7f69b40aeaf0 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69b0001690 con 0x7f69b40aeaf0 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 5 monclient: authenticate success, global_id 4159 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: dump: 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.974 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69b0001870 con 0x7f69b40aeaf0 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 1 -- 172.21.15.32:0/788228843 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69b0001c00 con 0x7f69b40aeaf0 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 4 set_mon_vals no callback set 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69abfff640 10 monclient: dump: 2024-03-29T11:14:37.975 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 10 monclient: shutdown 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 1 -- 172.21.15.32:0/788228843 >> v1:172.21.15.32:6789/0 conn(0x7f69b40aeaf0 legacy=0x7f69b40b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 1 -- 172.21.15.32:0/788228843 shutdown_connections 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 1 -- 172.21.15.32:0/788228843 >> 172.21.15.32:0/788228843 conn(0x7f69b40b7b60 msgr2=0x7f69b40b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.984+0000 7f69bb77b640 1 -- 172.21.15.32:0/788228843 shutdown_connections 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 -- 172.21.15.32:0/788228843 wait complete. 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 10 monclient: monmap: 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.977 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:37.978 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:37.980 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 librados: starting msgr at 2024-03-29T11:14:37.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 librados: starting objecter 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 Processor -- start 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 -- start start 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 librados: setting wanted keys 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 1 librados: calling monclient init 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.985+0000 7f69bb77b640 10 monclient: init 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-c con 0x7f69b4071f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-a con 0x7f69b40af520 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:37.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient(hunting): picked mon.noname-b con 0x7f69b4140670 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b412e640 con 0x7f69b40af520 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b4141810 con 0x7f69b4140670 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b4140150 con 0x7f69b4071f20 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69bb77b640 10 monclient(hunting): authenticate will time out at 801.249354s 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69b9d18640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f69b4140670 0x7f69b412df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:34844/0 (socket says 172.21.15.32:34844) 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69b9d18640 1 -- 172.21.15.32:0/2936088218 learned_addr learned my addr 172.21.15.32:0/2936088218 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter ms_handle_connect 0x7f69b4140670 2024-03-29T11:14:37.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter ms_handle_connect 0x7f69b4071f20 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter ms_handle_connect 0x7f69b40af520 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703460561 0 0) 0x7f69b4141810 con 0x7f69b4140670 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): my global_id is 4162 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6998004eb0 con 0x7f69b4140670 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974474728 0 0) 0x7f69b4140150 con 0x7f69b4071f20 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b4141810 con 0x7f69b4071f20 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1384255940 0 0) 0x7f69b412e640 con 0x7f69b40af520 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.986+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b4140150 con 0x7f69b40af520 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1845433590 0 0) 0x7f6998004eb0 con 0x7f69b4140670 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b412e640 con 0x7f69b4140670 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69ac001c30 con 0x7f69b4140670 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.985 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387944747 0 0) 0x7f69b4141810 con 0x7f69b4071f20 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6998004eb0 con 0x7f69b4071f20 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69a40018b0 con 0x7f69b4071f20 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.986 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636896390 0 0) 0x7f69b4140150 con 0x7f69b40af520 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b4141810 con 0x7f69b40af520 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69b0001b80 con 0x7f69b40af520 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2667952779 0 0) 0x7f69b412e640 con 0x7f69b4140670 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 >> v1:172.21.15.3:6790/0 conn(0x7f69b4071f20 legacy=0x7f69b414f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 >> v1:172.21.15.3:6789/0 conn(0x7f69b40af520 legacy=0x7f69b412d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 monclient: found mon.b 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40ae590 con 0x7f69b4140670 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 5 monclient: authenticate success, global_id 4162 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 10 monclient: _renew_subs 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b414f960 con 0x7f69b4140670 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69ac003990 con 0x7f69b4140670 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f69ac003da0 con 0x7f69b4140670 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 monclient: dump: 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:37.990 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69aaffd640 4 set_mon_vals no callback set 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 10 monclient: _renew_subs 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f69ac004310 con 0x7f69b4140670 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b413e4a0 con 0x7f69b4140670 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69bb77b640 1 librados: init done 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2815611475 0 0) 0x7f69ac0045c0 con 0x7f69b4140670 2024-03-29T11:14:37.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 10 client.4162.objecter ms_dispatch 0x7f69b4000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.987+0000 7f69a9ffb640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.988+0000 7f69bb77b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.988+0000 7f69bb77b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.988+0000 7f69bb77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.988+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b41402e0 con 0x7f69b4140670 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.990+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f69ac004780 con 0x7f69b4140670 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.990+0000 7f69a9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:37.990+0000 7f69a9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:38.035 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.058+0000 7f5c76038040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:14:38.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.063+0000 7f69bb77b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9a2d4629-bd65-420b-855e-3ca59aa5c06f", "id": 7}] 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.063+0000 7f69bb77b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9a2d4629-bd65-420b-855e-3ca59aa5c06f", "id": 7}] 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.063+0000 7f69bb77b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.063+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9a2d4629-bd65-420b-855e-3ca59aa5c06f", "id": 7} v 0) v1 -- 0x7f69b406a3f0 con 0x7f69b4140670 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.067+0000 7f69a9ffb640 1 -- 172.21.15.32:0/2936088218 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a2d4629-bd65-420b-855e-3ca59aa5c06f", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3595003559 0 1371665406) 0x7f69ac005b40 con 0x7f69b4140670 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.067+0000 7f69a9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9a2d4629-bd65-420b-855e-3ca59aa5c06f", "id": 7}] 2024-03-29T11:14:38.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.067+0000 7f69a9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:7 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 10 librados: watch_flush enter 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 10 librados: watch_flush exit 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 10 client.4162.objecter successfully canceled tick 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 10 monclient: shutdown 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 >> v1:172.21.15.32:6789/0 conn(0x7f69b4140670 legacy=0x7f69b412df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 shutdown_connections 2024-03-29T11:14:38.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 >> 172.21.15.32:0/2936088218 conn(0x7f69b40b7b60 msgr2=0x7f69b40b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:38.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 shutdown_connections 2024-03-29T11:14:38.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 -- 172.21.15.32:0/2936088218 wait complete. 2024-03-29T11:14:38.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-29T11:14:38.068+0000 7f69bb77b640 1 librados: shutdown 2024-03-29T11:14:38.057 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:14:38.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:14:38.059 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:14:38.060 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:14:38.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:14:38.062 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:14:38.062 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:14:38.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:14:38.065 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:14:38.065 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:14:38.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:14:38.068 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:14:38.069 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:14:38.069 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:14:38.069 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.092+0000 7f2eac012040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:14:38.071 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:14:38.071 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:14:38.071 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:14:38.073 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:14:38.074 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:14:38.074 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:14:38.076 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:14:38.076 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:14:38.076 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:14:38.078 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:14:38.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:38.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.221+0000 7f054f189640 10 monclient: get_monmap_and_config 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.221+0000 7f054f189640 10 monclient: build_initial_monmap 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.221+0000 7f054f189640 10 monclient: monmap: 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:14:38.222122+0000 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:14:38.222122+0000 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.199 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:38.200 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:38.200 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.221+0000 7f054f189640 1 Processor -- start 2024-03-29T11:14:38.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 1 -- start start 2024-03-29T11:14:38.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient: init 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient(hunting): picked mon.noname-b con 0x7f05480b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient(hunting): picked mon.noname-a con 0x7f05480a9ef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient(hunting): picked mon.noname-c con 0x7f0548146350 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0548145790 con 0x7f05480a9ef0 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480aabe0 con 0x7f0548146350 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480aaec0 con 0x7f05480b4a00 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f0547fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f05480a9ef0 0x7f05480b1950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:43616/0 (socket says 172.21.15.3:43616) 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054f189640 10 monclient(hunting): authenticate will time out at 805.476742s 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f0547fff640 1 -- 172.21.15.3:0/1180720620 learned_addr learned my addr 172.21.15.3:0/1180720620 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:38.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.222+0000 7f054cf25640 1 --1- 172.21.15.3:0/1180720620 >> v1:172.21.15.3:6790/0 conn(0x7f05480b4a00 0x7f0548145c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47220/0 (socket says 172.21.15.3:47220) 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617746462 0 0) 0x7f05480aaec0 con 0x7f05480b4a00 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0538003520 con 0x7f05480b4a00 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514502131 0 0) 0x7f0548145790 con 0x7f05480a9ef0 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05480aaec0 con 0x7f05480a9ef0 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10602046 0 0) 0x7f05480aabe0 con 0x7f0548146350 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0548145790 con 0x7f0548146350 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3394197380 0 0) 0x7f0538003520 con 0x7f05480b4a00 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 >> v1:172.21.15.32:6789/0 conn(0x7f0548146350 legacy=0x7f054814da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 >> v1:172.21.15.3:6789/0 conn(0x7f05480a9ef0 legacy=0x7f05480b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 monclient: found mon.noname-b 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0548142170 con 0x7f05480b4a00 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:38.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f053c001690 con 0x7f05480b4a00 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: dump: 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.211 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f053c0017f0 con 0x7f05480b4a00 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 1 -- 172.21.15.3:0/1180720620 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f053c001c00 con 0x7f05480b4a00 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 4 set_mon_vals no callback set 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f05477fe640 10 monclient: dump: 2024-03-29T11:14:38.212 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.213 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 10 monclient: shutdown 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 1 -- 172.21.15.3:0/1180720620 >> v1:172.21.15.3:6790/0 conn(0x7f05480b4a00 legacy=0x7f0548145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 1 -- 172.21.15.3:0/1180720620 shutdown_connections 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.223+0000 7f054f189640 1 -- 172.21.15.3:0/1180720620 >> 172.21.15.3:0/1180720620 conn(0x7f0548066920 msgr2=0x7f0548066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 -- 172.21.15.3:0/1180720620 shutdown_connections 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 -- 172.21.15.3:0/1180720620 wait complete. 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 10 monclient: build_initial_monmap 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 10 monclient: monmap: 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 0 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 0.000000 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:created 0.000000 2024-03-29T11:14:38.214 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 librados: starting msgr at 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 librados: starting objecter 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 Processor -- start 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 -- start start 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 librados: setting wanted keys 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.224+0000 7f054f189640 1 librados: calling monclient init 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient: init 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:14:38.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient(hunting): picked mon.noname-b con 0x7f05480a9ef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient(hunting): picked mon.noname-c con 0x7f05480b4a00 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f0547fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f05480b4a00 0x7f05480b0860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47224/0 (socket says 172.21.15.3:47224) 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054cf25640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f05480a9ef0 0x7f05480ae570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:59746/0 (socket says 172.21.15.3:59746) 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient(hunting): picked mon.noname-a con 0x7f0548146350 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480ae0e0 con 0x7f0548146350 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054d726640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0548146350 0x7f05481e1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:43622/0 (socket says 172.21.15.3:43622) 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f0547fff640 1 -- 172.21.15.3:0/1973419114 learned_addr learned my addr 172.21.15.3:0/1973419114 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480afb10 con 0x7f05480a9ef0 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05480afdf0 con 0x7f05480b4a00 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f054f189640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.225+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f05480b4a00 2024-03-29T11:14:38.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f05480a9ef0 2024-03-29T11:14:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 monclient(hunting): authenticate will time out at 805.480015s 2024-03-29T11:14:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 client.?.objecter ms_handle_connect 0x7f0548146350 2024-03-29T11:14:38.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:38.217 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f5c76038040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:14:38.218 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.235+0000 7f2eac012040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:14:38.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588209984 0 0) 0x7f05480afb10 con 0x7f05480a9ef0 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): my global_id is 4192 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c004eb0 con 0x7f05480a9ef0 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894118925 0 0) 0x7f05480afdf0 con 0x7f05480b4a00 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05480afb10 con 0x7f05480b4a00 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910979049 0 0) 0x7f05480ae0e0 con 0x7f0548146350 2024-03-29T11:14:38.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05480afdf0 con 0x7f0548146350 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 52165577 0 0) 0x7f05480afb10 con 0x7f05480b4a00 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05480ae0e0 con 0x7f05480b4a00 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0534001930 con 0x7f05480b4a00 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): dump: 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.222 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.223 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.223 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.223 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.223 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3779576251 0 0) 0x7f05480afdf0 con 0x7f0548146350 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05480afb10 con 0x7f0548146350 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 642906005 0 0) 0x7f052c004eb0 con 0x7f05480a9ef0 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05480afdf0 con 0x7f05480a9ef0 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0540001b50 con 0x7f0548146350 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): dump: 2024-03-29T11:14:38.224 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f053c001b00 con 0x7f05480a9ef0 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:38.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient(hunting): dump: 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1339662276 0 0) 0x7f05480ae0e0 con 0x7f05480b4a00 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 >> v1:172.21.15.32:6789/0 conn(0x7f05480a9ef0 legacy=0x7f05480ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 >> v1:172.21.15.3:6789/0 conn(0x7f0548146350 legacy=0x7f05481e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 monclient: found mon.c 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480aec80 con 0x7f05480b4a00 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f05457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 5 monclient: authenticate success, global_id 4184 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 monclient: _renew_subs 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05480aeed0 con 0x7f05480b4a00 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 monclient: _renew_subs 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05480b1150 con 0x7f05480b4a00 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.226+0000 7f054f189640 1 librados: init done 2024-03-29T11:14:38.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05340047e0 con 0x7f05480b4a00 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0534004af0 con 0x7f05480b4a00 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 10 monclient: dump: 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:epoch 1 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0534004fe0 con 0x7f05480b4a00 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f0534006870 con 0x7f05480b4a00 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f0546ffd640 4 set_mon_vals no callback set 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 10 client.4184.objecter ms_dispatch 0x7f0548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:38.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.227+0000 7f05457fa640 20 client.4184.objecter dump_active .. 0 homeless 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.228+0000 7f054f189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.228+0000 7f054f189640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.228+0000 7f054f189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.228+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0528005080 con 0x7f05480b4a00 2024-03-29T11:14:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.230+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0534006aa0 con 0x7f05480b4a00 2024-03-29T11:14:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.230+0000 7f05457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:38.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.230+0000 7f05457fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:38.251 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.274+0000 7f5c76038040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:14:38.260 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.282+0000 7f2eac012040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.291+0000 7f054f189640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.291+0000 7f054f189640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.291+0000 7f054f189640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.291+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0528005330 con 0x7f05480b4a00 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.292+0000 7f05457fa640 1 -- 172.21.15.3:0/1973419114 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f05340062b0 con 0x7f05480b4a00 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.292+0000 7f05457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.292+0000 7f05457fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:38.271 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-29T11:14:38.271 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 10 librados: watch_flush enter 2024-03-29T11:14:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 10 librados: watch_flush exit 2024-03-29T11:14:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 20 client.4184.objecter shutdown clearing up homeless session... 2024-03-29T11:14:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 10 client.4184.objecter successfully canceled tick 2024-03-29T11:14:38.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 10 monclient: shutdown 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 >> v1:172.21.15.3:6790/0 conn(0x7f05480b4a00 legacy=0x7f05480b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 shutdown_connections 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 >> 172.21.15.3:0/1973419114 conn(0x7f0548066920 msgr2=0x7f05480b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 shutdown_connections 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 -- 172.21.15.3:0/1973419114 wait complete. 2024-03-29T11:14:38.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-03-29T11:14:38.293+0000 7f054f189640 1 librados: shutdown 2024-03-29T11:14:38.281 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:14:38.281 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:14:38.369 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.391+0000 7f5c76038040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:14:38.376 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.399+0000 7f2eac012040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:14:38.467 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.490+0000 7f5c76038040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:14:38.476 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.498+0000 7f2eac012040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:14:38.627 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.650+0000 7f5c76038040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:14:38.637 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.659+0000 7f2eac012040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:14:38.671 INFO:tasks.ceph.osd.6.smithi032.stderr:2024-03-29T11:14:38.693+0000 7fbd474f9640 -1 Falling back to public interface 2024-03-29T11:14:38.671 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.694+0000 7f5c76038040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:14:38.672 INFO:tasks.ceph.osd.5.smithi032.stderr:2024-03-29T11:14:38.695+0000 7f0e4fad1640 -1 Falling back to public interface 2024-03-29T11:14:38.676 INFO:tasks.ceph.osd.7.smithi032.stderr:2024-03-29T11:14:38.698+0000 7fefa91fa640 -1 Falling back to public interface 2024-03-29T11:14:38.677 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-03-29T11:14:38.700+0000 7f2cb500d640 -1 Falling back to public interface 2024-03-29T11:14:38.678 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-03-29T11:14:38.701+0000 7fabe640c640 -1 Falling back to public interface 2024-03-29T11:14:38.683 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-03-29T11:14:38.705+0000 7fb06cb1f640 -1 Falling back to public interface 2024-03-29T11:14:38.684 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.706+0000 7f2eac012040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:14:38.690 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-03-29T11:14:38.710+0000 7f79f7c15640 -1 Falling back to public interface 2024-03-29T11:14:38.695 INFO:tasks.ceph.osd.4.smithi032.stderr:2024-03-29T11:14:38.718+0000 7f6142b9e640 -1 Falling back to public interface 2024-03-29T11:14:38.832 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:38.854+0000 7f5c76038040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:14:38.848 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:38.871+0000 7f2eac012040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:14:39.087 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:39.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:39.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 10 monclient: monmap: 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:14:39.200047+0000 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:14:39.200047+0000 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:39.177 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:39.178 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:39.178 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 1 Processor -- start 2024-03-29T11:14:39.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 1 -- start start 2024-03-29T11:14:39.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.199+0000 7f2a08d6b640 10 monclient: init 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a040b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a040b2cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a040b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040aab60 con 0x7f2a040b2cb0 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040aae40 con 0x7f2a040b28d0 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a040ab120 con 0x7f2a040b3560 2024-03-29T11:14:39.188 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a08d6b640 10 monclient(hunting): authenticate will time out at 806.454616s 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a01d9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2a040b2cb0 0x7f2a04142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:43628/0 (socket says 172.21.15.3:43628) 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.200+0000 7f2a01d9b640 1 -- 172.21.15.3:0/155829497 learned_addr learned my addr 172.21.15.3:0/155829497 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472057152 0 0) 0x7f2a040aab60 con 0x7f2a040b2cb0 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec003520 con 0x7f2a040b2cb0 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1999695551 0 0) 0x7f2a040ab120 con 0x7f2a040b3560 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.189 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a040aab60 con 0x7f2a040b3560 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047764916 0 0) 0x7f2a040aae40 con 0x7f2a040b28d0 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient(hunting): my global_id is 4195 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a040ab120 con 0x7f2a040b28d0 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863175965 0 0) 0x7f2a040aab60 con 0x7f2a040b3560 2024-03-29T11:14:39.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 >> v1:172.21.15.32:6789/0 conn(0x7f2a040b28d0 legacy=0x7f2a0413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 >> v1:172.21.15.3:6789/0 conn(0x7f2a040b2cb0 legacy=0x7f2a04142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 monclient: found mon.noname-b 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0413c380 con 0x7f2a040b3560 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29fc001970 con 0x7f2a040b3560 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:39.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: dump: 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29fc001b50 con 0x7f2a040b3560 2024-03-29T11:14:39.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 1 -- 172.21.15.3:0/155829497 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29fc001fa0 con 0x7f2a040b3560 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a0159a640 10 monclient: dump: 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.193 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 4 set_mon_vals no callback set 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 10 monclient: shutdown 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.201+0000 7f2a08d6b640 1 -- 172.21.15.3:0/155829497 >> v1:172.21.15.3:6790/0 conn(0x7f2a040b3560 legacy=0x7f2a041428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 1 -- 172.21.15.3:0/155829497 shutdown_connections 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 1 -- 172.21.15.3:0/155829497 >> 172.21.15.3:0/155829497 conn(0x7f2a04066a00 msgr2=0x7f2a04066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 1 -- 172.21.15.3:0/155829497 shutdown_connections 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 1 -- 172.21.15.3:0/155829497 wait complete. 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 10 monclient: build_initial_monmap 2024-03-29T11:14:39.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.202+0000 7f2a08d6b640 10 monclient: monmap: 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 librados: starting msgr at 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 librados: starting objecter 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 Processor -- start 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 -- start start 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 librados: setting wanted keys 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 librados: calling monclient init 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient: init 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f2a040b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f2a041444b0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f2a041df880 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a041462a0 con 0x7f2a041444b0 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a041d4cc0 con 0x7f2a040b28d0 2024-03-29T11:14:39.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a041d7ea0 con 0x7f2a041df880 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a08d6b640 10 monclient(hunting): authenticate will time out at 806.457902s 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a01d9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2a041444b0 0x7f2a04144be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:43638/0 (socket says 172.21.15.3:43638) 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a01d9b640 1 -- 172.21.15.3:0/3816001825 learned_addr learned my addr 172.21.15.3:0/3816001825 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.203+0000 7f2a02d9d640 1 --1- 172.21.15.3:0/3816001825 >> v1:172.21.15.3:6790/0 conn(0x7f2a041df880 0x7f2a041e1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47246/0 (socket says 172.21.15.3:47246) 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter ms_handle_connect 0x7f2a041df880 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter ms_handle_connect 0x7f2a041444b0 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter ms_handle_connect 0x7f2a040b28d0 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679301872 0 0) 0x7f2a041d7ea0 con 0x7f2a041df880 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e4004eb0 con 0x7f2a041df880 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152377955 0 0) 0x7f2a041462a0 con 0x7f2a041444b0 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a041d7ea0 con 0x7f2a041444b0 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126388422 0 0) 0x7f2a041d4cc0 con 0x7f2a040b28d0 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:39.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): my global_id is 4198 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a041462a0 con 0x7f2a040b28d0 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1980531942 0 0) 0x7f29e4004eb0 con 0x7f2a041df880 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a041d4cc0 con 0x7f2a041df880 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29fc001ee0 con 0x7f2a041df880 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.199 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3621035946 0 0) 0x7f2a041462a0 con 0x7f2a040b28d0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29e4004eb0 con 0x7f2a040b28d0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2574738017 0 0) 0x7f2a041d7ea0 con 0x7f2a041444b0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a041462a0 con 0x7f2a041444b0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29f8001870 con 0x7f2a040b28d0 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:39.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29f4001210 con 0x7f2a041444b0 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:39.201 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3049120518 0 0) 0x7f2a041d4cc0 con 0x7f2a041df880 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 >> v1:172.21.15.32:6789/0 conn(0x7f2a040b28d0 legacy=0x7f2a041d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 >> v1:172.21.15.3:6789/0 conn(0x7f2a041444b0 legacy=0x7f2a04144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:39.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 monclient: found mon.c 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a041452f0 con 0x7f2a041df880 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:14:39.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 10 monclient: _renew_subs 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041d7270 con 0x7f2a041df880 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 10 monclient: _renew_subs 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041e3570 con 0x7f2a041df880 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a08d6b640 1 librados: init done 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29fc0020e0 con 0x7f2a041df880 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f29fc003dc0 con 0x7f2a041df880 2024-03-29T11:14:39.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f29f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 10 monclient: dump: 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:39.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.204+0000 7f2a00d99640 4 set_mon_vals no callback set 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f29fc004370 con 0x7f2a041df880 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f29fc006d40 con 0x7f2a041df880 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 10 client.4190.objecter ms_dispatch 0x7f2a04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f29f2ffd640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f2a08d6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f2a08d6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f2a08d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.205+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e8005080 con 0x7f2a041df880 2024-03-29T11:14:39.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.207+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f29fc006670 con 0x7f2a041df880 2024-03-29T11:14:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.207+0000 7f29f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:39.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.207+0000 7f29f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f2a08d6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f2a08d6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f2a08d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29e8005330 con 0x7f2a041df880 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f29f2ffd640 1 -- 172.21.15.3:0/3816001825 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f29fc006810 con 0x7f2a041df880 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f29f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:39.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.270+0000 7f29f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:39.249 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:14:39.249 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:39.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 10 librados: watch_flush enter 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 10 librados: watch_flush exit 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 10 monclient: shutdown 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 >> v1:172.21.15.3:6790/0 conn(0x7f2a041df880 legacy=0x7f2a041e1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 shutdown_connections 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 >> 172.21.15.3:0/3816001825 conn(0x7f2a04066a00 msgr2=0x7f2a04135730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 shutdown_connections 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 -- 172.21.15.3:0/3816001825 wait complete. 2024-03-29T11:14:39.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:39.271+0000 7f2a08d6b640 1 librados: shutdown 2024-03-29T11:14:39.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:39.667 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:39.690+0000 7f5c76038040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:14:39.720 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:39.742+0000 7f2eac012040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:14:39.953 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:39.975+0000 7f5c76038040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:14:40.006 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:40.029+0000 7f5c76038040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:14:40.007 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:40.029+0000 7f2eac012040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:14:40.060 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:40.083+0000 7f2eac012040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:14:40.125 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:40.148+0000 7f5c76038040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:14:40.182 INFO:tasks.ceph.mgr.x.smithi032.stderr:2024-03-29T11:14:40.204+0000 7f2eac012040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:14:41.141 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:41.164+0000 7f5c16c3d640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:14:41.142 INFO:tasks.ceph.mgr.y.smithi003.stderr:2024-03-29T11:14:41.165+0000 7f5c16c3d640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:14:42.124 INFO:tasks.ceph.osd.2.smithi003.stderr:2024-03-29T11:14:42.147+0000 7f2cb500d640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:14:42.143 INFO:tasks.ceph.osd.5.smithi032.stderr:2024-03-29T11:14:42.165+0000 7f0e4fad1640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:14:42.187 INFO:tasks.ceph.osd.3.smithi003.stderr:2024-03-29T11:14:42.209+0000 7fabe640c640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:14:42.199 INFO:tasks.ceph.osd.1.smithi003.stderr:2024-03-29T11:14:42.222+0000 7f79f7c15640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:14:42.200 INFO:tasks.ceph.osd.0.smithi003.stderr:2024-03-29T11:14:42.223+0000 7fb06cb1f640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:14:42.204 INFO:tasks.ceph.osd.7.smithi032.stderr:2024-03-29T11:14:42.226+0000 7fefa91fa640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:14:42.223 INFO:tasks.ceph.osd.6.smithi032.stderr:2024-03-29T11:14:42.245+0000 7fbd474f9640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:14:42.224 INFO:tasks.ceph.osd.4.smithi032.stderr:2024-03-29T11:14:42.246+0000 7f6142b9e640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:14:46.066 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.236+0000 7f794a76d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:46.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.236+0000 7f794a76d640 10 monclient: build_initial_monmap 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.236+0000 7f794a76d640 10 monclient: monmap: 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:14:46.237426+0000 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:14:46.237426+0000 2024-03-29T11:14:46.215 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:46.216 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.216 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:46.216 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:46.216 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:46.216 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.238+0000 7f794a76d640 1 Processor -- start 2024-03-29T11:14:46.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.238+0000 7f794a76d640 1 -- start start 2024-03-29T11:14:46.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.238+0000 7f794a76d640 10 monclient: init 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.239+0000 7f794a76d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.239+0000 7f794a76d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.239+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f793c0389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f793c038dc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f793c030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c0c54c0 con 0x7f793c0389e0 2024-03-29T11:14:46.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c036430 con 0x7f793c038dc0 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c039840 con 0x7f793c030660 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794a76d640 10 monclient(hunting): authenticate will time out at 813.494065s 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794976b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f793c0389e0 0x7f793c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34912/0 (socket says 172.21.15.3:34912) 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.240+0000 7f794976b640 1 -- 172.21.15.3:0/3611188574 learned_addr learned my addr 172.21.15.3:0/3611188574 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047589662 0 0) 0x7f793c0c54c0 con 0x7f793c0389e0 2024-03-29T11:14:46.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:14:46.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7930003520 con 0x7f793c0389e0 2024-03-29T11:14:46.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103757325 0 0) 0x7f793c039840 con 0x7f793c030660 2024-03-29T11:14:46.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:14:46.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793c0c54c0 con 0x7f793c030660 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498352935 0 0) 0x7f793c036430 con 0x7f793c038dc0 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793c039840 con 0x7f793c038dc0 2024-03-29T11:14:46.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628044408 0 0) 0x7f7930003520 con 0x7f793c0389e0 2024-03-29T11:14:46.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 >> v1:172.21.15.3:6790/0 conn(0x7f793c030660 legacy=0x7f793c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 >> v1:172.21.15.32:6789/0 conn(0x7f793c038dc0 legacy=0x7f793c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:46.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 monclient: found mon.noname-a 2024-03-29T11:14:46.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c035690 con 0x7f793c0389e0 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: _finish_auth 0 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:46.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f794a76d640 5 monclient: authenticate success, global_id 4236 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f794a76d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f79340015c0 con 0x7f793c0389e0 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: dump: 2024-03-29T11:14:46.244 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.245 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f794a76d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 1 -- 172.21.15.3:0/3611188574 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79340017e0 con 0x7f793c0389e0 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f793bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:46.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f794a76d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.241+0000 7f794a76d640 4 set_mon_vals no callback set 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.242+0000 7f794a76d640 10 monclient: shutdown 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.242+0000 7f794a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.242+0000 7f794a76d640 1 -- 172.21.15.3:0/3611188574 >> v1:172.21.15.3:6789/0 conn(0x7f793c0389e0 legacy=0x7f793c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.242+0000 7f794a76d640 1 -- 172.21.15.3:0/3611188574 shutdown_connections 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.242+0000 7f794a76d640 1 -- 172.21.15.3:0/3611188574 >> 172.21.15.3:0/3611188574 conn(0x7f793c005c00 msgr2=0x7f793c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.243+0000 7f794a76d640 1 -- 172.21.15.3:0/3611188574 shutdown_connections 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.243+0000 7f794a76d640 1 -- 172.21.15.3:0/3611188574 wait complete. 2024-03-29T11:14:46.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 10 monclient: build_initial_monmap 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 10 monclient: monmap: 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 librados: starting msgr at 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 librados: starting objecter 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 Processor -- start 2024-03-29T11:14:46.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 -- start start 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 librados: setting wanted keys 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 1 librados: calling monclient init 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.244+0000 7f794a76d640 10 monclient: init 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-a con 0x7f793c030660 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-b con 0x7f793c0389e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient(hunting): picked mon.noname-c con 0x7f793c032650 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c0ca1c0 con 0x7f793c030660 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c0c96e0 con 0x7f793c0389e0 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793c0c9de0 con 0x7f793c032650 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.245+0000 7f794a76d640 10 monclient(hunting): authenticate will time out at 813.499807s 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7948f6a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f793c0389e0 0x7f793c031f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:38842/0 (socket says 172.21.15.3:38842) 2024-03-29T11:14:46.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7948f6a640 1 -- 172.21.15.3:0/2022653226 learned_addr learned my addr 172.21.15.3:0/2022653226 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter ms_handle_connect 0x7f793c032650 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter ms_handle_connect 0x7f793c0389e0 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter ms_handle_connect 0x7f793c030660 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388290670 0 0) 0x7f793c0c9de0 con 0x7f793c032650 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.246+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7928004eb0 con 0x7f793c032650 2024-03-29T11:14:46.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678944110 0 0) 0x7f793c0ca1c0 con 0x7f793c030660 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f793c0c9de0 con 0x7f793c030660 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3324563607 0 0) 0x7f7928004eb0 con 0x7f793c032650 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f793c0ca1c0 con 0x7f793c032650 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f794404fdd0 con 0x7f793c032650 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:46.252 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934120584 0 0) 0x7f793c0c96e0 con 0x7f793c0389e0 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:14:46.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7928004eb0 con 0x7f793c0389e0 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358031729 0 0) 0x7f793c0c9de0 con 0x7f793c030660 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f793c0c96e0 con 0x7f793c030660 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7934004980 con 0x7f793c030660 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.247+0000 7f7939ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:46.254 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2782456786 0 0) 0x7f7928004eb0 con 0x7f793c0389e0 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f793c0c9de0 con 0x7f793c0389e0 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578868219 0 0) 0x7f793c0ca1c0 con 0x7f793c032650 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 >> v1:172.21.15.32:6789/0 conn(0x7f793c0389e0 legacy=0x7f793c031f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 >> v1:172.21.15.3:6789/0 conn(0x7f793c030660 legacy=0x7f793c0ca650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 monclient: found mon.c 2024-03-29T11:14:46.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793c032a80 con 0x7f793c032650 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f7939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 5 monclient: authenticate success, global_id 4232 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 10 monclient: _renew_subs 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793c164520 con 0x7f793c032650 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 10 client.4232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 10 monclient: _renew_subs 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f793c1649b0 con 0x7f793c032650 2024-03-29T11:14:46.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.248+0000 7f794a76d640 1 librados: init done 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f794a76d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f794a76d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f794a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793c007b70 con 0x7f793c032650 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f794404f430 con 0x7f793c032650 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f794404eee0 con 0x7f793c032650 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f7939ffb640 10 monclient: dump: 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:46.257 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.250+0000 7f793b7fe640 4 set_mon_vals no callback set 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.254+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f7944067be0 con 0x7f793c032650 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f79440fe0c0 con 0x7f793c032650 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f794976b640 10 monclient: get_auth_request con 0x7f792806de90 auth_method 0 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 10 client.4232.objecter ms_dispatch 0x7f7944000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 3 client.4232.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 3 client.4232.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 20 client.4232.objecter dump_active .. 0 homeless 2024-03-29T11:14:46.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.255+0000 7f7939ffb640 10 client.4232.objecter ms_handle_connect 0x7f792806de90 2024-03-29T11:14:46.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.260+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79440c9a00 con 0x7f793c032650 2024-03-29T11:14:46.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.260+0000 7f7939ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:46.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.260+0000 7f7939ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:46.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.710+0000 7f794a76d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:46.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.710+0000 7f794a76d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:46.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.710+0000 7f794a76d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:46.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.710+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f793c0c9f70 con 0x7f793c032650 2024-03-29T11:14:46.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.712+0000 7f7939ffb640 1 -- 172.21.15.3:0/2022653226 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f794404f040 con 0x7f793c032650 2024-03-29T11:14:46.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.712+0000 7f7939ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:46.690 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:14:46.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.712+0000 7f7939ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:46.690 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 10 librados: watch_flush enter 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 10 librados: watch_flush exit 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 20 client.4232.objecter shutdown clearing up homeless session... 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 10 client.4232.objecter successfully canceled tick 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 >> v1:172.21.15.3:6816/844689376 conn(0x7f792806de90 legacy=0x7f7928070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 10 monclient: shutdown 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 >> v1:172.21.15.3:6790/0 conn(0x7f793c032650 legacy=0x7f793c162e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:46.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 shutdown_connections 2024-03-29T11:14:46.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 >> 172.21.15.3:0/2022653226 conn(0x7f793c005c00 msgr2=0x7f793c0c1840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:46.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 shutdown_connections 2024-03-29T11:14:46.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 -- 172.21.15.3:0/2022653226 wait complete. 2024-03-29T11:14:46.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:46.716+0000 7f794a76d640 1 librados: shutdown 2024-03-29T11:14:46.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:53.518 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.691+0000 7feace9d6640 10 monclient: get_monmap_and_config 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.691+0000 7feace9d6640 10 monclient: build_initial_monmap 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.691+0000 7feace9d6640 10 monclient: monmap: 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:14:53.692725+0000 2024-03-29T11:14:53.669 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:14:53.692725+0000 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:14:53.670 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.692+0000 7feace9d6640 1 Processor -- start 2024-03-29T11:14:53.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.693+0000 7feace9d6640 1 -- start start 2024-03-29T11:14:53.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.693+0000 7feace9d6640 10 monclient: init 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.693+0000 7feace9d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.693+0000 7feace9d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-c con 0x7feac80b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-a con 0x7feac80b2cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feac7fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feac80b28d0 0x7feac813bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:43300/0 (socket says 172.21.15.3:43300) 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feac7fff640 1 -- 172.21.15.3:0/2517556913 learned_addr learned my addr 172.21.15.3:0/2517556913 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-b con 0x7feac80b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80aab60 con 0x7feac80b2cb0 2024-03-29T11:14:53.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80aae40 con 0x7feac80b28d0 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac80ab120 con 0x7feac80b3560 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.694+0000 7feace9d6640 10 monclient(hunting): authenticate will time out at 820.948825s 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447021381 0 0) 0x7feac80ab120 con 0x7feac80b3560 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003520 con 0x7feac80b3560 2024-03-29T11:14:53.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695881633 0 0) 0x7feac80aae40 con 0x7feac80b28d0 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac80ab120 con 0x7feac80b28d0 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408700498 0 0) 0x7feac80aab60 con 0x7feac80b2cb0 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac80aae40 con 0x7feac80b2cb0 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046806111 0 0) 0x7feab0003520 con 0x7feac80b3560 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 >> v1:172.21.15.32:6789/0 conn(0x7feac80b28d0 legacy=0x7feac813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 >> v1:172.21.15.3:6789/0 conn(0x7feac80b2cb0 legacy=0x7feac8142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.695+0000 7feac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac813c380 con 0x7feac80b3560 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feac0001a10 con 0x7feac80b3560 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: dump: 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:53.678 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feace9d6640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feace9d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac0001b70 con 0x7feac80b3560 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feace9d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:53.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feace9d6640 4 set_mon_vals no callback set 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 1 -- 172.21.15.3:0/2517556913 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feac0001f80 con 0x7feac80b3560 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.696+0000 7feac6ffd640 10 monclient: dump: 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:53.680 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.697+0000 7feace9d6640 10 monclient: shutdown 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.697+0000 7feace9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.697+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 >> v1:172.21.15.3:6790/0 conn(0x7feac80b3560 legacy=0x7feac81428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.697+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 shutdown_connections 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.697+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 >> 172.21.15.3:0/2517556913 conn(0x7feac8066a00 msgr2=0x7feac8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.698+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 shutdown_connections 2024-03-29T11:14:53.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.698+0000 7feace9d6640 1 -- 172.21.15.3:0/2517556913 wait complete. 2024-03-29T11:14:53.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.699+0000 7feace9d6640 10 monclient: build_initial_monmap 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.699+0000 7feace9d6640 10 monclient: monmap: 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.699+0000 7feace9d6640 1 librados: starting msgr at 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.699+0000 7feace9d6640 1 librados: starting objecter 2024-03-29T11:14:53.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.699+0000 7feace9d6640 1 Processor -- start 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.700+0000 7feace9d6640 1 -- start start 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.700+0000 7feace9d6640 1 librados: setting wanted keys 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.700+0000 7feace9d6640 1 librados: calling monclient init 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.700+0000 7feace9d6640 10 monclient: init 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-b con 0x7feac80b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-a con 0x7feac813e400 addr v1:172.21.15.3:6789/0 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 10 monclient(hunting): picked mon.noname-c con 0x7feac81dff60 addr v1:172.21.15.3:6790/0 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.701+0000 7feace9d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac8144fe0 con 0x7feac813e400 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feace9d6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac81365c0 con 0x7feac80b28d0 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feace9d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac81368a0 con 0x7feac81dff60 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feace9d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:53.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feace9d6640 10 monclient(hunting): authenticate will time out at 820.956251s 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feaccf73640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feac81dff60 0x7feac81e2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:46034/0 (socket says 172.21.15.3:46034) 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feaccf73640 1 -- 172.21.15.3:0/1890832717 learned_addr learned my addr 172.21.15.3:0/1890832717 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feac4ff9640 10 client.?.objecter ms_handle_connect 0x7feac81dff60 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feac4ff9640 10 client.?.objecter ms_handle_connect 0x7feac813e400 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.702+0000 7feac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745601341 0 0) 0x7feac81368a0 con 0x7feac81dff60 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab4004eb0 con 0x7feac81dff60 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551566982 0 0) 0x7feac8144fe0 con 0x7feac813e400 2024-03-29T11:14:53.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feac81368a0 con 0x7feac813e400 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4131981168 0 0) 0x7feab4004eb0 con 0x7feac81dff60 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feac8144fe0 con 0x7feac81dff60 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feac0001f80 con 0x7feac81dff60 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.690 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.703+0000 7feac4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 123751218 0 0) 0x7feac81368a0 con 0x7feac813e400 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feab4004eb0 con 0x7feac813e400 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feabc0018b0 con 0x7feac813e400 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient(hunting): dump: 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:53.691 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1706129629 0 0) 0x7feac8144fe0 con 0x7feac81dff60 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 >> v1:172.21.15.32:6789/0 conn(0x7feac80b28d0 legacy=0x7feac81457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 >> v1:172.21.15.3:6789/0 conn(0x7feac813e400 legacy=0x7feac8137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 monclient: found mon.c 2024-03-29T11:14:53.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8136b80 con 0x7feac81dff60 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.704+0000 7feac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 5 monclient: authenticate success, global_id 4238 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 10 monclient: _renew_subs 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac81452a0 con 0x7feac81dff60 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 10 monclient: _renew_subs 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feac0003c20 con 0x7feac81dff60 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.705+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feac81e3a70 con 0x7feac81dff60 2024-03-29T11:14:53.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feace9d6640 1 librados: init done 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feac0003fb0 con 0x7feac81dff60 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac4ff9640 10 monclient: dump: 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:14:53.694 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.706+0000 7feac67fc640 4 set_mon_vals no callback set 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.707+0000 7feace9d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.707+0000 7feace9d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.707+0000 7feace9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.707+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac806a400 con 0x7feac81dff60 2024-03-29T11:14:53.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.709+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7feac0004190 con 0x7feac81dff60 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.710+0000 7feac7fff640 10 monclient: get_auth_request con 0x7feab406de30 auth_method 0 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.710+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7feac0084f00 con 0x7feac81dff60 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.710+0000 7feac4ff9640 10 client.4238.objecter ms_dispatch 0x7feac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.710+0000 7feac4ff9640 3 client.4238.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.710+0000 7feac4ff9640 3 client.4238.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.711+0000 7feac4ff9640 20 client.4238.objecter dump_active .. 0 homeless 2024-03-29T11:14:53.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.711+0000 7feac4ff9640 10 client.4238.objecter ms_handle_connect 0x7feab406de30 2024-03-29T11:14:53.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.719+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feac0050840 con 0x7feac81dff60 2024-03-29T11:14:53.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.719+0000 7feac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:53.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:53.719+0000 7feac4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.084+0000 7feace9d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.084+0000 7feace9d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.084+0000 7feace9d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.084+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feac8145430 con 0x7feac81dff60 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.085+0000 7feac4ff9640 1 -- 172.21.15.3:0/1890832717 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7feac00845b0 con 0x7feac81dff60 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.085+0000 7feac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:54.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.085+0000 7feac4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:54.063 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:14:54.063 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.088+0000 7feace9d6640 10 librados: watch_flush enter 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 10 librados: watch_flush exit 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 20 client.4238.objecter shutdown clearing up homeless session... 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 10 client.4238.objecter successfully canceled tick 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 >> v1:172.21.15.3:6816/844689376 conn(0x7feab406de30 legacy=0x7feab40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 10 monclient: shutdown 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:54.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 >> v1:172.21.15.3:6790/0 conn(0x7feac81dff60 legacy=0x7feac81e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:54.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 shutdown_connections 2024-03-29T11:14:54.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 >> 172.21.15.3:0/1890832717 conn(0x7feac8066a00 msgr2=0x7feac813b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:54.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 shutdown_connections 2024-03-29T11:14:54.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 -- 172.21.15.3:0/1890832717 wait complete. 2024-03-29T11:14:54.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:14:54.089+0000 7feace9d6640 1 librados: shutdown 2024-03-29T11:14:54.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:00.889 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.061+0000 7ffab6daf640 10 monclient: get_monmap_and_config 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.061+0000 7ffab6daf640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.061+0000 7ffab6daf640 10 monclient: monmap: 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:01.039 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:01.062565+0000 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:01.062565+0000 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.062+0000 7ffab6daf640 1 Processor -- start 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.062+0000 7ffab6daf640 1 -- start start 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.062+0000 7ffab6daf640 10 monclient: init 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-b con 0x7ffab00b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:01.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-c con 0x7ffab00b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-a con 0x7ffab00b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00aab60 con 0x7ffab00b3560 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00aae40 con 0x7ffab00b2cb0 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab00ab120 con 0x7ffab00b28d0 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.063+0000 7ffab6daf640 10 monclient(hunting): authenticate will time out at 828.317729s 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.064+0000 7ffaaffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffab00b2cb0 0x7ffab0142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:35836/0 (socket says 172.21.15.3:35836) 2024-03-29T11:15:01.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.064+0000 7ffaaffff640 1 -- 172.21.15.3:0/3303805042 learned_addr learned my addr 172.21.15.3:0/3303805042 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930734905 0 0) 0x7ffab00aae40 con 0x7ffab00b2cb0 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa98003520 con 0x7ffab00b2cb0 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620756339 0 0) 0x7ffab00ab120 con 0x7ffab00b28d0 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab00aae40 con 0x7ffab00b28d0 2024-03-29T11:15:01.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453638248 0 0) 0x7ffa98003520 con 0x7ffab00b2cb0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 >> v1:172.21.15.3:6790/0 conn(0x7ffab00b28d0 legacy=0x7ffab013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 >> v1:172.21.15.3:6789/0 conn(0x7ffab00b3560 legacy=0x7ffab01428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab013c380 con 0x7ffab00b2cb0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffaa0001690 con 0x7ffab00b2cb0 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.066+0000 7ffaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:01.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: dump: 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffab6daf640 5 monclient: authenticate success, global_id 4246 2024-03-29T11:15:01.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffab6daf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaa00017f0 con 0x7ffab00b2cb0 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 1 -- 172.21.15.3:0/3303805042 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffaa0001c40 con 0x7ffab00b2cb0 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffaaf7fe640 10 monclient: dump: 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.050 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffab6daf640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.067+0000 7ffab6daf640 4 set_mon_vals no callback set 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.068+0000 7ffab6daf640 10 monclient: shutdown 2024-03-29T11:15:01.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.068+0000 7ffab6daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:01.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.068+0000 7ffab6daf640 1 -- 172.21.15.3:0/3303805042 >> v1:172.21.15.32:6789/0 conn(0x7ffab00b2cb0 legacy=0x7ffab0142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.068+0000 7ffab6daf640 1 -- 172.21.15.3:0/3303805042 shutdown_connections 2024-03-29T11:15:01.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.068+0000 7ffab6daf640 1 -- 172.21.15.3:0/3303805042 >> 172.21.15.3:0/3303805042 conn(0x7ffab0066a00 msgr2=0x7ffab0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:01.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.069+0000 7ffab6daf640 1 -- 172.21.15.3:0/3303805042 shutdown_connections 2024-03-29T11:15:01.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.069+0000 7ffab6daf640 1 -- 172.21.15.3:0/3303805042 wait complete. 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.070+0000 7ffab6daf640 10 monclient: build_initial_monmap 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.070+0000 7ffab6daf640 10 monclient: monmap: 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:01.054 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.070+0000 7ffab6daf640 1 librados: starting msgr at 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.070+0000 7ffab6daf640 1 librados: starting objecter 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.071+0000 7ffab6daf640 1 Processor -- start 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.071+0000 7ffab6daf640 1 -- start start 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.071+0000 7ffab6daf640 1 librados: setting wanted keys 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.071+0000 7ffab6daf640 1 librados: calling monclient init 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.071+0000 7ffab6daf640 10 monclient: init 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.072+0000 7ffab6daf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.072+0000 7ffab6daf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.072+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-c con 0x7ffab00b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-b con 0x7ffab013ec10 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 10 monclient(hunting): picked mon.noname-a con 0x7ffab0132180 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:01.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab006e0d0 con 0x7ffab0132180 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab006e3b0 con 0x7ffab013ec10 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab0132710 con 0x7ffab00b28d0 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.074+0000 7ffab6daf640 10 monclient(hunting): authenticate will time out at 828.328392s 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.075+0000 7ffab4b4b640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ffab00b28d0 0x7ffab013e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:36971/0 (socket says 172.21.15.3:36971) 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.075+0000 7ffab4b4b640 1 -- 172.21.15.3:0/4119575318 learned_addr learned my addr 172.21.15.3:0/4119575318 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.075+0000 7ffaad7fa640 10 client.?.objecter ms_handle_connect 0x7ffab00b28d0 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.075+0000 7ffaad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798289214 0 0) 0x7ffab0132710 con 0x7ffab00b28d0 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:15:01.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c004eb0 con 0x7ffab00b28d0 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374329091 0 0) 0x7ffa9c004eb0 con 0x7ffab00b28d0 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffab0132710 con 0x7ffab00b28d0 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffaa8001dd0 con 0x7ffab00b28d0 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:01.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:01.058 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:01.059 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:01.059 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.076+0000 7ffaad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 client.?.objecter ms_handle_connect 0x7ffab013ec10 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144022478 0 0) 0x7ffab006e3b0 con 0x7ffab013ec10 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:01.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.077+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c0056d0 con 0x7ffab013ec10 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 client.?.objecter ms_handle_connect 0x7ffab0132180 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3735471565 0 0) 0x7ffab0132710 con 0x7ffab00b28d0 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 >> v1:172.21.15.32:6789/0 conn(0x7ffab013ec10 legacy=0x7ffab013c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 >> v1:172.21.15.3:6789/0 conn(0x7ffab0132180 legacy=0x7ffab013cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 1 monclient: found mon.c 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab006e7d0 con 0x7ffab00b28d0 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:01.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffaad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 10 monclient: _renew_subs 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab013d450 con 0x7ffab00b28d0 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 10 monclient: _renew_subs 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab01456a0 con 0x7ffab00b28d0 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.078+0000 7ffab6daf640 1 librados: init done 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffaa80020d0 con 0x7ffab00b28d0 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:01.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffaa8001dd0 con 0x7ffab00b28d0 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.079+0000 7ffaad7fa640 10 monclient: dump: 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.080+0000 7ffaae7fc640 4 set_mon_vals no callback set 2024-03-29T11:15:01.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.080+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ffaa8003c40 con 0x7ffab00b28d0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7ffaa80548b0 con 0x7ffab00b28d0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffaad7fa640 10 client.4244.objecter ms_dispatch 0x7ffab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffaad7fa640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffaad7fa640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffaad7fa640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffab6daf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffab6daf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffab6daf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.081+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa90005080 con 0x7ffab00b28d0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.082+0000 7ffaaffff640 10 monclient: get_auth_request con 0x7ffa9c06df40 auth_method 0 2024-03-29T11:15:01.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.082+0000 7ffaad7fa640 10 client.4244.objecter ms_handle_connect 0x7ffa9c06df40 2024-03-29T11:15:01.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.091+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffaa8050b50 con 0x7ffab00b28d0 2024-03-29T11:15:01.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.091+0000 7ffaad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:01.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.091+0000 7ffaad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:01.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.427+0000 7ffab6daf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:01.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.427+0000 7ffab6daf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:01.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.427+0000 7ffab6daf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:01.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.427+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa90005330 con 0x7ffab00b28d0 2024-03-29T11:15:01.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.428+0000 7ffaad7fa640 1 -- 172.21.15.3:0/4119575318 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7ffaa8050cf0 con 0x7ffab00b28d0 2024-03-29T11:15:01.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.428+0000 7ffaad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:01.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.428+0000 7ffaad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:01.406 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:01.406 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 10 librados: watch_flush enter 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 10 librados: watch_flush exit 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 >> v1:172.21.15.3:6816/844689376 conn(0x7ffa9c06df40 legacy=0x7ffa9c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 10 monclient: shutdown 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 >> v1:172.21.15.3:6790/0 conn(0x7ffab00b28d0 legacy=0x7ffab013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 shutdown_connections 2024-03-29T11:15:01.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.432+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 >> 172.21.15.3:0/4119575318 conn(0x7ffab0066a00 msgr2=0x7ffab0134d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:01.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.433+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 shutdown_connections 2024-03-29T11:15:01.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.433+0000 7ffab6daf640 1 -- 172.21.15.3:0/4119575318 wait complete. 2024-03-29T11:15:01.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:01.433+0000 7ffab6daf640 1 librados: shutdown 2024-03-29T11:15:01.428 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:08.236 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:08.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.407+0000 7f8027382640 10 monclient: get_monmap_and_config 2024-03-29T11:15:08.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.407+0000 7f8027382640 10 monclient: build_initial_monmap 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.407+0000 7f8027382640 10 monclient: monmap: 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:08.408700+0000 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:08.408700+0000 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:08.386 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.409+0000 7f8027382640 1 Processor -- start 2024-03-29T11:15:08.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.409+0000 7f8027382640 1 -- start start 2024-03-29T11:15:08.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.409+0000 7f8027382640 10 monclient: init 2024-03-29T11:15:08.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.410+0000 7f8027382640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:08.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.410+0000 7f8027382640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:08.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.410+0000 7f8027382640 10 monclient(hunting): picked mon.noname-b con 0x7f802006efb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:08.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 10 monclient(hunting): picked mon.noname-a con 0x7f8020072450 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 10 monclient(hunting): picked mon.noname-c con 0x7f80201389c0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200714e0 con 0x7f8020072450 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200717c0 con 0x7f80201389c0 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8020071f30 con 0x7f802006efb0 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.413+0000 7f8027382640 10 monclient(hunting): authenticate will time out at 835.667107s 2024-03-29T11:15:08.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.414+0000 7f802511e640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f802006efb0 0x7f802013ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54421/0 (socket says 172.21.15.3:54421) 2024-03-29T11:15:08.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.414+0000 7f802591f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f80201389c0 0x7f802014a620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:33284/0 (socket says 172.21.15.3:33284) 2024-03-29T11:15:08.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.414+0000 7f802591f640 1 -- 172.21.15.3:0/3846669473 learned_addr learned my addr 172.21.15.3:0/3846669473 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008745996 0 0) 0x7f80200717c0 con 0x7f80201389c0 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f800c003520 con 0x7f80201389c0 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603857367 0 0) 0x7f8020071f30 con 0x7f802006efb0 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:08.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.416+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80200717c0 con 0x7f802006efb0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335116870 0 0) 0x7f800c003520 con 0x7f80201389c0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 >> v1:172.21.15.3:6790/0 conn(0x7f802006efb0 legacy=0x7f802013ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 >> v1:172.21.15.3:6789/0 conn(0x7f8020072450 legacy=0x7f802006f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 monclient: found mon.noname-c 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80200728d0 con 0x7f80201389c0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:08.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8010001690 con 0x7f80201389c0 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: dump: 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80100017f0 con 0x7f80201389c0 2024-03-29T11:15:08.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 1 -- 172.21.15.3:0/3846669473 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8010001b80 con 0x7f80201389c0 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.417+0000 7f8017fff640 10 monclient: dump: 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.418+0000 7f8027382640 5 monclient: authenticate success, global_id 4252 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.418+0000 7f8027382640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.418+0000 7f8027382640 4 set_mon_vals no callback set 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.419+0000 7f8027382640 10 monclient: shutdown 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.419+0000 7f8027382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.419+0000 7f8027382640 1 -- 172.21.15.3:0/3846669473 >> v1:172.21.15.32:6789/0 conn(0x7f80201389c0 legacy=0x7f802014a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.419+0000 7f8027382640 1 -- 172.21.15.3:0/3846669473 shutdown_connections 2024-03-29T11:15:08.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.419+0000 7f8027382640 1 -- 172.21.15.3:0/3846669473 >> 172.21.15.3:0/3846669473 conn(0x7f80200674a0 msgr2=0x7f80200b4c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.420+0000 7f8027382640 1 -- 172.21.15.3:0/3846669473 shutdown_connections 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.420+0000 7f8027382640 1 -- 172.21.15.3:0/3846669473 wait complete. 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.421+0000 7f8027382640 10 monclient: build_initial_monmap 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.421+0000 7f8027382640 10 monclient: monmap: 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:08.404 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 librados: starting msgr at 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 librados: starting objecter 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 Processor -- start 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 -- start start 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 librados: setting wanted keys 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 1 librados: calling monclient init 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.422+0000 7f8027382640 10 monclient: init 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.423+0000 7f8027382640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.423+0000 7f8027382640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.423+0000 7f8027382640 10 monclient(hunting): picked mon.noname-b con 0x7f802006efb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.423+0000 7f8027382640 10 monclient(hunting): picked mon.noname-a con 0x7f8020072450 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.423+0000 7f8027382640 10 monclient(hunting): picked mon.noname-c con 0x7f80201389c0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:08.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.424+0000 7f8027382640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200aac00 con 0x7f8020072450 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.424+0000 7f8027382640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80200aae50 con 0x7f802006efb0 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.424+0000 7f8027382640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802013ba90 con 0x7f80201389c0 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.424+0000 7f8027382640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8027382640 10 monclient(hunting): authenticate will time out at 835.679000s 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f802511e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f802006efb0 0x7f80200a9710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:33934/0 (socket says 172.21.15.3:33934) 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f802511e640 1 -- 172.21.15.3:0/455921782 learned_addr learned my addr 172.21.15.3:0/455921782 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 client.?.objecter ms_handle_connect 0x7f802006efb0 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195898951 0 0) 0x7f80200aae50 con 0x7f802006efb0 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.425+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004004eb0 con 0x7f802006efb0 2024-03-29T11:15:08.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2982256860 0 0) 0x7f8004004eb0 con 0x7f802006efb0 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80200aae50 con 0x7f802006efb0 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8018001bc0 con 0x7f802006efb0 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:08.407 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 client.?.objecter ms_handle_connect 0x7f80201389c0 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 client.?.objecter ms_handle_connect 0x7f8020072450 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354691947 0 0) 0x7f802013ba90 con 0x7f80201389c0 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.426+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80040056d0 con 0x7f80201389c0 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444651071 0 0) 0x7f80200aac00 con 0x7f8020072450 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:08.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:08.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:15:08.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80040059f0 con 0x7f8020072450 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445092490 0 0) 0x7f80200aae50 con 0x7f802006efb0 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 >> v1:172.21.15.3:6790/0 conn(0x7f80201389c0 legacy=0x7f80201d2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 >> v1:172.21.15.3:6789/0 conn(0x7f8020072450 legacy=0x7f8020135e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 monclient: found mon.b 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802013bc20 con 0x7f802006efb0 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 5 monclient: authenticate success, global_id 4255 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 10 monclient: _renew_subs 2024-03-29T11:15:08.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8020136580 con 0x7f802006efb0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 10 monclient: _renew_subs 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80201d3680 con 0x7f802006efb0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.427+0000 7f8027382640 1 librados: init done 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8018001e20 con 0x7f802006efb0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8018004e10 con 0x7f802006efb0 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.428+0000 7f8015ffb640 10 monclient: dump: 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:08.412 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.429+0000 7f8027382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.429+0000 7f8027382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.429+0000 7f8027382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.429+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802006a400 con 0x7f802006efb0 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f8018005030 con 0x7f802006efb0 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f80177fe640 4 set_mon_vals no callback set 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f8018084e10 con 0x7f802006efb0 2024-03-29T11:15:08.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 10 client.4255.objecter ms_dispatch 0x7f8020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:08.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:08.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:08.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.430+0000 7f8015ffb640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-29T11:15:08.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.431+0000 7f802491d640 10 monclient: get_auth_request con 0x7f800406e350 auth_method 0 2024-03-29T11:15:08.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.431+0000 7f8015ffb640 10 client.4255.objecter ms_handle_connect 0x7f800406e350 2024-03-29T11:15:08.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.438+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80180506f0 con 0x7f802006efb0 2024-03-29T11:15:08.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.438+0000 7f8015ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:08.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.438+0000 7f8015ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:08.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.778+0000 7f8027382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:08.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.778+0000 7f8027382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:08.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.778+0000 7f8027382640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:08.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.778+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80201d2bd0 con 0x7f802006efb0 2024-03-29T11:15:08.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.779+0000 7f8015ffb640 1 -- 172.21.15.3:0/455921782 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f8018052620 con 0x7f802006efb0 2024-03-29T11:15:08.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.779+0000 7f8015ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:08.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.779+0000 7f8015ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:08.757 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:08.757 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:08.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.783+0000 7f8027382640 10 librados: watch_flush enter 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 10 librados: watch_flush exit 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 10 client.4255.objecter successfully canceled tick 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 >> v1:172.21.15.3:6816/844689376 conn(0x7f800406e350 legacy=0x7f8004070810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 10 monclient: shutdown 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 >> v1:172.21.15.32:6789/0 conn(0x7f802006efb0 legacy=0x7f80200a9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 shutdown_connections 2024-03-29T11:15:08.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.784+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 >> 172.21.15.3:0/455921782 conn(0x7f80200674a0 msgr2=0x7f8020138240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:08.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.785+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 shutdown_connections 2024-03-29T11:15:08.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.785+0000 7f8027382640 1 -- 172.21.15.3:0/455921782 wait complete. 2024-03-29T11:15:08.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:08.785+0000 7f8027382640 1 librados: shutdown 2024-03-29T11:15:08.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:15.582 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.753+0000 7f1f1b989640 10 monclient: get_monmap_and_config 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.753+0000 7f1f1b989640 10 monclient: build_initial_monmap 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.753+0000 7f1f1b989640 10 monclient: monmap: 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.731 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:15.754311+0000 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:15.754311+0000 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:15.732 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.733 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.756+0000 7f1f1b989640 1 Processor -- start 2024-03-29T11:15:15.733 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.756+0000 7f1f1b989640 1 -- start start 2024-03-29T11:15:15.733 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.756+0000 7f1f1b989640 10 monclient: init 2024-03-29T11:15:15.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-b con 0x7f1f14072320 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-a con 0x7f1f140a9ec0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-c con 0x7f1f14142ba0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1413fbc0 con 0x7f1f140a9ec0 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f140af890 con 0x7f1f14142ba0 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f14141f90 con 0x7f1f14072320 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.757+0000 7f1f1b989640 10 monclient(hunting): authenticate will time out at 843.011623s 2024-03-29T11:15:15.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.761+0000 7f1f18f24640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1f140a9ec0 0x7f1f14142490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44687/0 (socket says 172.21.15.3:44687) 2024-03-29T11:15:15.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f18f24640 1 -- 172.21.15.3:0/2156901937 learned_addr learned my addr 172.21.15.3:0/2156901937 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342326738 0 0) 0x7f1f1413fbc0 con 0x7f1f140a9ec0 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.762+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f00003520 con 0x7f1f140a9ec0 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538902542 0 0) 0x7f1f00003520 con 0x7f1f140a9ec0 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 >> v1:172.21.15.3:6790/0 conn(0x7f1f14072320 legacy=0x7f1f140af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 >> v1:172.21.15.32:6789/0 conn(0x7f1f14142ba0 legacy=0x7f1f1414d8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 monclient: found mon.noname-a 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f140a9a70 con 0x7f1f140a9ec0 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: _finish_auth 0 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1f10001690 con 0x7f1f140a9ec0 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.763+0000 7f1f0bfff640 10 monclient: dump: 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.743 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:15.744 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f100017f0 con 0x7f1f140a9ec0 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 1 -- 172.21.15.3:0/2156901937 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1f10001c40 con 0x7f1f140a9ec0 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f0bfff640 10 monclient: dump: 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:15.746 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f1b989640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f1b989640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.766+0000 7f1f1b989640 4 set_mon_vals no callback set 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.767+0000 7f1f1b989640 10 monclient: shutdown 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.767+0000 7f1f1b989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.767+0000 7f1f1b989640 1 -- 172.21.15.3:0/2156901937 >> v1:172.21.15.3:6789/0 conn(0x7f1f140a9ec0 legacy=0x7f1f14142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.768+0000 7f1f1b989640 1 -- 172.21.15.3:0/2156901937 shutdown_connections 2024-03-29T11:15:15.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.768+0000 7f1f1b989640 1 -- 172.21.15.3:0/2156901937 >> 172.21.15.3:0/2156901937 conn(0x7f1f140b3c10 msgr2=0x7f1f140b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:15.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.769+0000 7f1f1b989640 1 -- 172.21.15.3:0/2156901937 shutdown_connections 2024-03-29T11:15:15.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.769+0000 7f1f1b989640 1 -- 172.21.15.3:0/2156901937 wait complete. 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.770+0000 7f1f1b989640 10 monclient: build_initial_monmap 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.770+0000 7f1f1b989640 10 monclient: monmap: 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 librados: starting msgr at 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 librados: starting objecter 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 Processor -- start 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 -- start start 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 librados: setting wanted keys 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 1 librados: calling monclient init 2024-03-29T11:15:15.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 10 monclient: init 2024-03-29T11:15:15.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:15.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.771+0000 7f1f1b989640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:15.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-a con 0x7f1f14072320 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-b con 0x7f1f140a9ec0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 10 monclient(hunting): picked mon.noname-c con 0x7f1f14142ba0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f141369c0 con 0x7f1f14072320 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f14070820 con 0x7f1f140a9ec0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f1406fa10 con 0x7f1f14142ba0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.772+0000 7f1f1b989640 10 monclient(hunting): authenticate will time out at 843.026063s 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.777+0000 7f1f19725640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1f14072320 0x7f1f14135a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44801/0 (socket says 172.21.15.3:44801) 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.777+0000 7f1f19725640 1 -- 172.21.15.3:0/1681823932 learned_addr learned my addr 172.21.15.3:0/1681823932 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 client.?.objecter ms_handle_connect 0x7f1f14142ba0 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 client.?.objecter ms_handle_connect 0x7f1f14072320 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:15.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083473670 0 0) 0x7f1f1406fa10 con 0x7f1f14142ba0 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ef8004eb0 con 0x7f1f14142ba0 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093522581 0 0) 0x7f1f141369c0 con 0x7f1f14072320 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.778+0000 7f1f09ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.779+0000 7f1f09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.779+0000 7f1f09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.779+0000 7f1f09ffb640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:15:15.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.779+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1406fa10 con 0x7f1f14072320 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.779+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888619322 0 0) 0x7f1ef8004eb0 con 0x7f1f14142ba0 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f141369c0 con 0x7f1f14142ba0 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2124220888 0 0) 0x7f1f1406fa10 con 0x7f1f14072320 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ef8004eb0 con 0x7f1f14072320 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1f04001750 con 0x7f1f14142ba0 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.763 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 client.?.objecter ms_handle_connect 0x7f1f140a9ec0 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1f0c001b50 con 0x7f1f14072320 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.764 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1298946784 0 0) 0x7f1f141369c0 con 0x7f1f14142ba0 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.780+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 >> v1:172.21.15.32:6789/0 conn(0x7f1f140a9ec0 legacy=0x7f1f14070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 >> v1:172.21.15.3:6789/0 conn(0x7f1f14072320 legacy=0x7f1f14135a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 1 monclient: found mon.c 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:15.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1406fcf0 con 0x7f1f14142ba0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 10 monclient: _renew_subs 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f14145fa0 con 0x7f1f14142ba0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 10 client.4253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 10 monclient: _renew_subs 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f140712c0 con 0x7f1f14142ba0 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f1b989640 1 librados: init done 2024-03-29T11:15:15.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f040018b0 con 0x7f1f14142ba0 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1f04001d60 con 0x7f1f14142ba0 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.781+0000 7f1f09ffb640 10 monclient: dump: 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:15.767 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:15.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.782+0000 7f1f0b7fe640 4 set_mon_vals no callback set 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.783+0000 7f1f1b989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.783+0000 7f1f1b989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.783+0000 7f1f1b989640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.783+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef4005080 con 0x7f1f14142ba0 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.786+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f1f04003790 con 0x7f1f14142ba0 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.787+0000 7f1f19725640 10 monclient: get_auth_request con 0x7f1ef806df00 auth_method 0 2024-03-29T11:15:15.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.788+0000 7f1f09ffb640 10 client.4253.objecter ms_handle_connect 0x7f1ef806df00 2024-03-29T11:15:15.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.788+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f1f040835f0 con 0x7f1f14142ba0 2024-03-29T11:15:15.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.788+0000 7f1f09ffb640 10 client.4253.objecter ms_dispatch 0x7f1f14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:15.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.788+0000 7f1f09ffb640 3 client.4253.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:15.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.788+0000 7f1f09ffb640 3 client.4253.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:15.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.789+0000 7f1f09ffb640 20 client.4253.objecter dump_active .. 0 homeless 2024-03-29T11:15:15.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.796+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f0404ed00 con 0x7f1f14142ba0 2024-03-29T11:15:15.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.796+0000 7f1f09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:15.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:15.796+0000 7f1f09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:16.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.136+0000 7f1f1b989640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.136+0000 7f1f1b989640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.136+0000 7f1f1b989640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:16.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.136+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ef4005330 con 0x7f1f14142ba0 2024-03-29T11:15:16.117 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:16.117 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:16.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.137+0000 7f1f09ffb640 1 -- 172.21.15.3:0/1681823932 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f1f04050e00 con 0x7f1f14142ba0 2024-03-29T11:15:16.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.138+0000 7f1f09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:16.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.138+0000 7f1f09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.141+0000 7f1f1b989640 10 librados: watch_flush enter 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.141+0000 7f1f1b989640 10 librados: watch_flush exit 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.141+0000 7f1f1b989640 20 client.4253.objecter shutdown clearing up homeless session... 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.141+0000 7f1f1b989640 10 client.4253.objecter successfully canceled tick 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 >> v1:172.21.15.3:6816/844689376 conn(0x7f1ef806df00 legacy=0x7f1ef80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 10 monclient: shutdown 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 >> v1:172.21.15.3:6790/0 conn(0x7f1f14142ba0 legacy=0x7f1f141ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 shutdown_connections 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 >> 172.21.15.3:0/1681823932 conn(0x7f1f140b3c10 msgr2=0x7f1f140b3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 shutdown_connections 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 -- 172.21.15.3:0/1681823932 wait complete. 2024-03-29T11:15:16.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:16.142+0000 7f1f1b989640 1 librados: shutdown 2024-03-29T11:15:16.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:22.948 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:23.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.122+0000 7fcb18819640 10 monclient: get_monmap_and_config 2024-03-29T11:15:23.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.122+0000 7fcb18819640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.122+0000 7fcb18819640 10 monclient: monmap: 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:23.123005+0000 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:23.123005+0000 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:23.100 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.123+0000 7fcb18819640 1 Processor -- start 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.123+0000 7fcb18819640 1 -- start start 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.123+0000 7fcb18819640 10 monclient: init 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.124+0000 7fcb18819640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.124+0000 7fcb18819640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.124+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c0389e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:23.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-c con 0x7fcb0c038dc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c030660 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0c54c0 con 0x7fcb0c030660 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c036430 con 0x7fcb0c038dc0 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb12ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb0c038dc0 0x7fcb0c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:41601/0 (socket says 172.21.15.3:41601) 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb12ffd640 1 -- 172.21.15.3:0/1426344635 learned_addr learned my addr 172.21.15.3:0/1426344635 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c039840 con 0x7fcb0c0389e0 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.127+0000 7fcb18819640 10 monclient(hunting): authenticate will time out at 850.381343s 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540772080 0 0) 0x7fcb0c036430 con 0x7fcb0c038dc0 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:15:23.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.128+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb04003520 con 0x7fcb0c038dc0 2024-03-29T11:15:23.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.129+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443057149 0 0) 0x7fcb04003520 con 0x7fcb0c038dc0 2024-03-29T11:15:23.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.129+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 >> v1:172.21.15.3:6790/0 conn(0x7fcb0c0389e0 legacy=0x7fcb0c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:23.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.129+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 >> v1:172.21.15.3:6789/0 conn(0x7fcb0c030660 legacy=0x7fcb0c0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:23.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 1 monclient: found mon.noname-c 2024-03-29T11:15:23.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c035690 con 0x7fcb0c038dc0 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fcb00001690 con 0x7fcb0c038dc0 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb127fc640 10 monclient: dump: 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.110 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb18819640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.130+0000 7fcb18819640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb00001860 con 0x7fcb0c038dc0 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 1 -- 172.21.15.3:0/1426344635 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fcb00001cb0 con 0x7fcb0c038dc0 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:23.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb127fc640 10 monclient: dump: 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:23.112 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb18819640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.132+0000 7fcb18819640 4 set_mon_vals no callback set 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 10 monclient: shutdown 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 >> v1:172.21.15.32:6789/0 conn(0x7fcb0c038dc0 legacy=0x7fcb0c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 shutdown_connections 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 >> 172.21.15.3:0/1426344635 conn(0x7fcb0c005c00 msgr2=0x7fcb0c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.133+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 shutdown_connections 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.134+0000 7fcb18819640 1 -- 172.21.15.3:0/1426344635 wait complete. 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.134+0000 7fcb18819640 10 monclient: build_initial_monmap 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.134+0000 7fcb18819640 10 monclient: monmap: 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:23.116 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 librados: starting msgr at 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 librados: starting objecter 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 Processor -- start 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 -- start start 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 librados: setting wanted keys 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 1 librados: calling monclient init 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.135+0000 7fcb18819640 10 monclient: init 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:23.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:23.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-b con 0x7fcb0c030660 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:23.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-a con 0x7fcb0c0389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:23.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient(hunting): picked mon.noname-c con 0x7fcb0c0c5d60 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:23.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0c7d00 con 0x7fcb0c0389e0 2024-03-29T11:15:23.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0c7a20 con 0x7fcb0c030660 2024-03-29T11:15:23.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0c0c6160 con 0x7fcb0c0c5d60 2024-03-29T11:15:23.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:23.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.136+0000 7fcb18819640 10 monclient(hunting): authenticate will time out at 850.390715s 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.140+0000 7fcb13fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fcb0c0c5d60 0x7fcb0c173e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:60879/0 (socket says 172.21.15.3:60879) 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.140+0000 7fcb137fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fcb0c030660 0x7fcb0c0c8120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:41677/0 (socket says 172.21.15.3:41677) 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.140+0000 7fcb13fff640 1 -- 172.21.15.3:0/4105682850 learned_addr learned my addr 172.21.15.3:0/4105682850 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 client.?.objecter ms_handle_connect 0x7fcb0c0c5d60 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 client.?.objecter ms_handle_connect 0x7fcb0c030660 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531792252 0 0) 0x7fcb0c0c7a20 con 0x7fcb0c030660 2024-03-29T11:15:23.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.141+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8004eb0 con 0x7fcb0c030660 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3581631435 0 0) 0x7fcaf8004eb0 con 0x7fcb0c030660 2024-03-29T11:15:23.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcb0c0c7a20 con 0x7fcb0c030660 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fcb08003fc0 con 0x7fcb0c030660 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 10 monclient(hunting): dump: 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:23.122 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.143+0000 7fcafffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030402635 0 0) 0x7fcb0c0c6160 con 0x7fcb0c0c5d60 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8004eb0 con 0x7fcb0c0c5d60 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 client.?.objecter ms_handle_connect 0x7fcb0c0389e0 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:23.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3275425951 0 0) 0x7fcb0c0c7a20 con 0x7fcb0c030660 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 >> v1:172.21.15.3:6790/0 conn(0x7fcb0c0c5d60 legacy=0x7fcb0c173e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 >> v1:172.21.15.3:6789/0 conn(0x7fcb0c0389e0 legacy=0x7fcb0c0c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 monclient: found mon.b 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c0c8830 con 0x7fcb0c030660 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.144+0000 7fcafffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:23.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 5 monclient: authenticate success, global_id 4264 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 10 monclient: _renew_subs 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c00c4d0 con 0x7fcb0c030660 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 10 monclient: _renew_subs 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c175550 con 0x7fcb0c030660 2024-03-29T11:15:23.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.145+0000 7fcb18819640 1 librados: init done 2024-03-29T11:15:23.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.147+0000 7fcb18819640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.147+0000 7fcb18819640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.147+0000 7fcb18819640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.147+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c007b90 con 0x7fcb0c030660 2024-03-29T11:15:23.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.149+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcb08004120 con 0x7fcb0c030660 2024-03-29T11:15:23.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.149+0000 7fcafffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:23.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.149+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fcb08001130 con 0x7fcb0c030660 2024-03-29T11:15:23.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.149+0000 7fcb11ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.150+0000 7fcafffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.150+0000 7fcafffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.150+0000 7fcafffff640 10 monclient: dump: 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:23.133 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.151+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fcb08001390 con 0x7fcb0c030660 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.151+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fcb080849d0 con 0x7fcb0c030660 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.154+0000 7fcb12ffd640 10 monclient: get_auth_request con 0x7fcaf806dd10 auth_method 0 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.155+0000 7fcafffff640 10 client.4264.objecter ms_dispatch 0x7fcb14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.155+0000 7fcafffff640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.155+0000 7fcafffff640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.155+0000 7fcafffff640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-29T11:15:23.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.155+0000 7fcafffff640 10 client.4264.objecter ms_handle_connect 0x7fcaf806dd10 2024-03-29T11:15:23.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.157+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcb080502b0 con 0x7fcb0c030660 2024-03-29T11:15:23.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.157+0000 7fcafffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:23.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.158+0000 7fcafffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:23.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.486+0000 7fcb18819640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:23.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.486+0000 7fcb18819640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:23.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.487+0000 7fcb18819640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:23.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.487+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb0c0c7840 con 0x7fcb0c030660 2024-03-29T11:15:23.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.489+0000 7fcafffff640 1 -- 172.21.15.3:0/4105682850 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fcb080521e0 con 0x7fcb0c030660 2024-03-29T11:15:23.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.490+0000 7fcafffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:23.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.490+0000 7fcafffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:23.467 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:23.467 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:23.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.493+0000 7fcb18819640 10 librados: watch_flush enter 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.493+0000 7fcb18819640 10 librados: watch_flush exit 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.493+0000 7fcb18819640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.493+0000 7fcb18819640 10 client.4264.objecter successfully canceled tick 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 >> v1:172.21.15.3:6816/844689376 conn(0x7fcaf806dd10 legacy=0x7fcaf80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 10 monclient: shutdown 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:23.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 >> v1:172.21.15.32:6789/0 conn(0x7fcb0c030660 legacy=0x7fcb0c0c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 shutdown_connections 2024-03-29T11:15:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.494+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 >> 172.21.15.3:0/4105682850 conn(0x7fcb0c005c00 msgr2=0x7fcb0c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.495+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 shutdown_connections 2024-03-29T11:15:23.472 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.495+0000 7fcb18819640 1 -- 172.21.15.3:0/4105682850 wait complete. 2024-03-29T11:15:23.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:23.495+0000 7fcb18819640 1 librados: shutdown 2024-03-29T11:15:23.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:30.298 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.472+0000 7f82b6ea8640 10 monclient: get_monmap_and_config 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.472+0000 7f82b6ea8640 10 monclient: build_initial_monmap 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.472+0000 7f82b6ea8640 10 monclient: monmap: 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:30.472953+0000 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:30.472953+0000 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:30.450 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:30.451 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:30.451 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:30.451 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.475+0000 7f82b6ea8640 1 Processor -- start 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.475+0000 7f82b6ea8640 1 -- start start 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.475+0000 7f82b6ea8640 10 monclient: init 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:30.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f82b0149c10 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f82b00a9630 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f82b014a450 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00aad40 con 0x7f82b0149c10 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b014a8e0 con 0x7f82b014a450 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00abb60 con 0x7f82b00a9630 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:30.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.476+0000 7f82b6ea8640 10 monclient(hunting): authenticate will time out at 857.730571s 2024-03-29T11:15:30.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.480+0000 7f82b4c44640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f82b0149c10 0x7f82b00ae7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56257/0 (socket says 172.21.15.3:56257) 2024-03-29T11:15:30.457 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.480+0000 7f82b4c44640 1 -- 172.21.15.3:0/3557507316 learned_addr learned my addr 172.21.15.3:0/3557507316 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:30.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174425491 0 0) 0x7f82b00abb60 con 0x7f82b00a9630 2024-03-29T11:15:30.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:30.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:30.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:30.464 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298003520 con 0x7f82b00a9630 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476063731 0 0) 0x7f82b00aad40 con 0x7f82b0149c10 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b00abb60 con 0x7f82b0149c10 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630844338 0 0) 0x7f82b014a8e0 con 0x7f82b014a450 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b00aad40 con 0x7f82b014a450 2024-03-29T11:15:30.465 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 123385531 0 0) 0x7f8298003520 con 0x7f82b00a9630 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 >> v1:172.21.15.32:6789/0 conn(0x7f82b014a450 legacy=0x7f82b0151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 >> v1:172.21.15.3:6789/0 conn(0x7f82b0149c10 legacy=0x7f82b00ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 monclient: found mon.noname-b 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00aa8d0 con 0x7f82b00a9630 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82b6ea8640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82b6ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f82a0001690 con 0x7f82b00a9630 2024-03-29T11:15:30.466 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.483+0000 7f82af7fe640 10 monclient: dump: 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:30.467 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.484+0000 7f82b6ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.484+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a0001870 con 0x7f82b00a9630 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.484+0000 7f82af7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.484+0000 7f82b6ea8640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.484+0000 7f82b6ea8640 4 set_mon_vals no callback set 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.485+0000 7f82b6ea8640 10 monclient: shutdown 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.485+0000 7f82b6ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.485+0000 7f82b6ea8640 1 -- 172.21.15.3:0/3557507316 >> v1:172.21.15.3:6790/0 conn(0x7f82b00a9630 legacy=0x7f82b0143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.485+0000 7f82af7fe640 1 -- 172.21.15.3:0/3557507316 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f82a0001690 con 0x7f82b00a9630 2024-03-29T11:15:30.468 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.485+0000 7f82af7fe640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.487+0000 7f82b6ea8640 1 -- 172.21.15.3:0/3557507316 shutdown_connections 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.488+0000 7f82b6ea8640 1 -- 172.21.15.3:0/3557507316 >> 172.21.15.3:0/3557507316 conn(0x7f82b00b3c10 msgr2=0x7f82b00b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.489+0000 7f82b6ea8640 1 -- 172.21.15.3:0/3557507316 shutdown_connections 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.490+0000 7f82b6ea8640 1 -- 172.21.15.3:0/3557507316 wait complete. 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.491+0000 7f82b6ea8640 10 monclient: build_initial_monmap 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.491+0000 7f82b6ea8640 10 monclient: monmap: 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:30.470 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 librados: starting msgr at 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 librados: starting objecter 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 Processor -- start 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 -- start start 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 librados: setting wanted keys 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 1 librados: calling monclient init 2024-03-29T11:15:30.471 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.492+0000 7f82b6ea8640 10 monclient: init 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-a con 0x7f82b00a9630 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-b con 0x7f82b0149c10 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient(hunting): picked mon.noname-c con 0x7f82b014a450 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00b3980 con 0x7f82b00a9630 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b0145fc0 con 0x7f82b0149c10 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00b1c40 con 0x7f82b014a450 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:30.473 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.494+0000 7f82b6ea8640 10 monclient(hunting): authenticate will time out at 857.748466s 2024-03-29T11:15:30.478 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.499+0000 7f82affff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f82b0149c10 0x7f82b01deed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:34063/0 (socket says 172.21.15.3:34063) 2024-03-29T11:15:30.479 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.499+0000 7f82affff640 1 -- 172.21.15.3:0/632702613 learned_addr learned my addr 172.21.15.3:0/632702613 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.500+0000 7f82ad7fa640 10 client.?.objecter ms_handle_connect 0x7f82b0149c10 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.500+0000 7f82ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090027754 0 0) 0x7f82b0145fc0 con 0x7f82b0149c10 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c004eb0 con 0x7f82b0149c10 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092634954 0 0) 0x7f829c004eb0 con 0x7f82b0149c10 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f82b0145fc0 con 0x7f82b0149c10 2024-03-29T11:15:30.480 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f82a00035f0 con 0x7f82b0149c10 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.501+0000 7f82ad7fa640 10 monclient(hunting): dump: 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:30.481 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 client.?.objecter ms_handle_connect 0x7f82b00a9630 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60163065 0 0) 0x7f82b00b3980 con 0x7f82b00a9630 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f829c0056d0 con 0x7f82b00a9630 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2213389713 0 0) 0x7f82b0145fc0 con 0x7f82b0149c10 2024-03-29T11:15:30.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.502+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 >> v1:172.21.15.3:6790/0 conn(0x7f82b014a450 legacy=0x7f82b01e5f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 >> v1:172.21.15.3:6789/0 conn(0x7f82b00a9630 legacy=0x7f82b01458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 1 monclient: found mon.b 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00b1dd0 con 0x7f82b0149c10 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 10 monclient: _finish_auth 0 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 10 monclient: _renew_subs 2024-03-29T11:15:30.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b01df5e0 con 0x7f82b0149c10 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 10 monclient: _renew_subs 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b0144e70 con 0x7f82b0149c10 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.506+0000 7f82b6ea8640 1 librados: init done 2024-03-29T11:15:30.485 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f8296ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f8296ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f8296ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8290005080 con 0x7f82b0149c10 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82a0003b00 con 0x7f82b0149c10 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f82a00035f0 con 0x7f82b0149c10 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ad7fa640 10 monclient: dump: 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:30.487 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:30.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.508+0000 7f82ae7fc640 4 set_mon_vals no callback set 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.510+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f82a0003c70 con 0x7f82b0149c10 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82b4c44640 10 monclient: get_auth_request con 0x7f829c06ded0 auth_method 0 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f82a00011e0 con 0x7f82b0149c10 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 10 client.4270.objecter ms_dispatch 0x7f82b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-29T11:15:30.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.511+0000 7f82ad7fa640 10 client.4270.objecter ms_handle_connect 0x7f829c06ded0 2024-03-29T11:15:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.516+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82a00508e0 con 0x7f82b0149c10 2024-03-29T11:15:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.516+0000 7f82ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:30.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.516+0000 7f82ad7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:30.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.852+0000 7f8296ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:30.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.852+0000 7f8296ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:30.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.852+0000 7f8296ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:30.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.852+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8290005330 con 0x7f82b0149c10 2024-03-29T11:15:30.832 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:30.833 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:30.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.854+0000 7f82ad7fa640 1 -- 172.21.15.3:0/632702613 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f82a0050a80 con 0x7f82b0149c10 2024-03-29T11:15:30.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.854+0000 7f82ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:30.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.854+0000 7f82ad7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:30.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.857+0000 7f8296ffd640 10 librados: watch_flush enter 2024-03-29T11:15:30.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 10 librados: watch_flush exit 2024-03-29T11:15:30.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-29T11:15:30.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 10 client.4270.objecter successfully canceled tick 2024-03-29T11:15:30.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 >> v1:172.21.15.3:6816/844689376 conn(0x7f829c06ded0 legacy=0x7f829c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 10 monclient: shutdown 2024-03-29T11:15:30.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:30.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 >> v1:172.21.15.32:6789/0 conn(0x7f82b0149c10 legacy=0x7f82b01deed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:30.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 shutdown_connections 2024-03-29T11:15:30.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.861+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 >> 172.21.15.3:0/632702613 conn(0x7f82b00b3c10 msgr2=0x7f82b00af080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:30.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.862+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 shutdown_connections 2024-03-29T11:15:30.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.862+0000 7f8296ffd640 1 -- 172.21.15.3:0/632702613 wait complete. 2024-03-29T11:15:30.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:30.864+0000 7f8296ffd640 1 librados: shutdown 2024-03-29T11:15:30.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:37.668 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:37.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.843+0000 7ff0c759e640 10 monclient: get_monmap_and_config 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.843+0000 7ff0c759e640 10 monclient: build_initial_monmap 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.843+0000 7ff0c759e640 10 monclient: monmap: 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:37.845260+0000 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:37.845260+0000 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:37.822 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.844+0000 7ff0c759e640 1 Processor -- start 2024-03-29T11:15:37.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.844+0000 7ff0c759e640 1 -- start start 2024-03-29T11:15:37.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.844+0000 7ff0c759e640 10 monclient: init 2024-03-29T11:15:37.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.845+0000 7ff0c759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:37.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.845+0000 7ff0c759e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:37.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.846+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c80b28c0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c80b39c0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c813c0e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80aaf90 con 0x7ff0c813c0e0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c81420c0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c8142cf0 con 0x7ff0c80b39c0 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c759e640 10 monclient(hunting): authenticate will time out at 865.103552s 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c659c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff0c80b28c0 0x7ff0c81425e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:35307/0 (socket says 172.21.15.3:35307) 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.848+0000 7ff0c659c640 1 -- 172.21.15.3:0/3263997004 learned_addr learned my addr 172.21.15.3:0/3263997004 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:37.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186220524 0 0) 0x7ff0c81420c0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.849+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0b4003520 con 0x7ff0c80b28c0 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.850+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745994544 0 0) 0x7ff0b4003520 con 0x7ff0c80b28c0 2024-03-29T11:15:37.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.850+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 >> v1:172.21.15.3:6790/0 conn(0x7ff0c80b39c0 legacy=0x7ff0c813b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:37.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 >> v1:172.21.15.3:6789/0 conn(0x7ff0c813c0e0 legacy=0x7ff0c814d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8142e80 con 0x7ff0c80b28c0 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.852+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b8001690 con 0x7ff0c80b28c0 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: dump: 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:37.832 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0b8001920 con 0x7ff0c80b28c0 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 1 -- 172.21.15.3:0/3263997004 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b8001b10 con 0x7ff0c80b28c0 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:37.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.853+0000 7ff0c5d9b640 10 monclient: dump: 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:37.834 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.835 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:37.835 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:37.835 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:37.835 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.855+0000 7ff0c759e640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.855+0000 7ff0c759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.855+0000 7ff0c759e640 4 set_mon_vals no callback set 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.856+0000 7ff0c759e640 10 monclient: shutdown 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.856+0000 7ff0c759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:37.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.856+0000 7ff0c759e640 1 -- 172.21.15.3:0/3263997004 >> v1:172.21.15.32:6789/0 conn(0x7ff0c80b28c0 legacy=0x7ff0c81425e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:37.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.857+0000 7ff0c759e640 1 -- 172.21.15.3:0/3263997004 shutdown_connections 2024-03-29T11:15:37.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.857+0000 7ff0c759e640 1 -- 172.21.15.3:0/3263997004 >> 172.21.15.3:0/3263997004 conn(0x7ff0c8066a00 msgr2=0x7ff0c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:37.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.859+0000 7ff0c759e640 1 -- 172.21.15.3:0/3263997004 shutdown_connections 2024-03-29T11:15:37.838 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.860+0000 7ff0c759e640 1 -- 172.21.15.3:0/3263997004 wait complete. 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.861+0000 7ff0c759e640 10 monclient: build_initial_monmap 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.861+0000 7ff0c759e640 10 monclient: monmap: 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:37.841 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.862+0000 7ff0c759e640 1 librados: starting msgr at 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.862+0000 7ff0c759e640 1 librados: starting objecter 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.862+0000 7ff0c759e640 1 Processor -- start 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.862+0000 7ff0c759e640 1 -- start start 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.863+0000 7ff0c759e640 1 librados: setting wanted keys 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.863+0000 7ff0c759e640 1 librados: calling monclient init 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.863+0000 7ff0c759e640 10 monclient: init 2024-03-29T11:15:37.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.864+0000 7ff0c759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:37.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.864+0000 7ff0c759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:37.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.864+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-a con 0x7ff0c80b28c0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-b con 0x7ff0c80b39c0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 10 monclient(hunting): picked mon.noname-c con 0x7ff0c813c0e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c81daf30 con 0x7ff0c80b28c0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80b7360 con 0x7ff0c80b39c0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c8144700 con 0x7ff0c813c0e0 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:37.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.868+0000 7ff0c759e640 10 monclient(hunting): authenticate will time out at 865.123125s 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.871+0000 7ff0c659c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff0c80b28c0 0x7ff0c81da820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54905/0 (socket says 172.21.15.3:54905) 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.871+0000 7ff0c659c640 1 -- 172.21.15.3:0/1731302568 learned_addr learned my addr 172.21.15.3:0/1731302568 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 client.?.objecter ms_handle_connect 0x7ff0c80b28c0 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945000334 0 0) 0x7ff0b8000fe0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:15:37.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.872+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a8004eb0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2459457808 0 0) 0x7ff0a8004eb0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0b8000fe0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b8001bb0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 10 monclient(hunting): dump: 2024-03-29T11:15:37.852 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 19012128 0 0) 0x7ff0b8000fe0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.873+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 >> v1:172.21.15.3:6790/0 conn(0x7ff0c813c0e0 legacy=0x7ff0c81e2320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 >> v1:172.21.15.32:6789/0 conn(0x7ff0c80b39c0 legacy=0x7ff0c8133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 1 monclient: found mon.a 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c80b75b0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 monclient: _finish_auth 0 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:37.855 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 client.?.objecter ms_handle_connect 0x7ff0c80b39c0 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.874+0000 7ff0beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 10 monclient: _renew_subs 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c80b6830 con 0x7ff0c80b28c0 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:37.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 10 monclient: _renew_subs 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c8143480 con 0x7ff0c80b28c0 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.875+0000 7ff0c759e640 1 librados: init done 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.879+0000 7ff0c759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.879+0000 7ff0c759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.879+0000 7ff0c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:15:37.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.879+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c806a400 con 0x7ff0c80b28c0 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff0b8004a90 con 0x7ff0c80b28c0 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff0b8004ee0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:37.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 10 monclient: dump: 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:37.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.880+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ff0b8005100 con 0x7ff0c80b28c0 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.882+0000 7ff0c559a640 4 set_mon_vals no callback set 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.883+0000 7ff0bffff640 10 monclient: get_auth_request con 0x7ff0a806dc10 auth_method 0 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.883+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7ff0b8084d90 con 0x7ff0c80b28c0 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.883+0000 7ff0beffd640 10 client.4269.objecter ms_dispatch 0x7ff0c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.883+0000 7ff0beffd640 3 client.4269.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.883+0000 7ff0beffd640 3 client.4269.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:37.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.884+0000 7ff0beffd640 20 client.4269.objecter dump_active .. 0 homeless 2024-03-29T11:15:37.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.884+0000 7ff0beffd640 10 client.4269.objecter ms_handle_connect 0x7ff0a806dc10 2024-03-29T11:15:37.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.888+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff0b80506f0 con 0x7ff0c80b28c0 2024-03-29T11:15:37.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.889+0000 7ff0beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:37.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:37.889+0000 7ff0beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0c759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0c759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0c759e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0c80b7740 con 0x7ff0c80b28c0 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0beffd640 1 -- 172.21.15.3:0/1731302568 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7ff0b8052620 con 0x7ff0c80b28c0 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:38.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.227+0000 7ff0beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:38.208 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:38.208 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 10 librados: watch_flush enter 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 10 librados: watch_flush exit 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 20 client.4269.objecter shutdown clearing up homeless session... 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 10 client.4269.objecter successfully canceled tick 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 >> v1:172.21.15.3:6816/844689376 conn(0x7ff0a806dc10 legacy=0x7ff0a80700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 10 monclient: shutdown 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 >> v1:172.21.15.3:6789/0 conn(0x7ff0c80b28c0 legacy=0x7ff0c81da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:38.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 shutdown_connections 2024-03-29T11:15:38.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.233+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 >> 172.21.15.3:0/1731302568 conn(0x7ff0c8066a00 msgr2=0x7ff0c8138990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:38.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.234+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 shutdown_connections 2024-03-29T11:15:38.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.234+0000 7ff0c759e640 1 -- 172.21.15.3:0/1731302568 wait complete. 2024-03-29T11:15:38.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:38.234+0000 7ff0c759e640 1 librados: shutdown 2024-03-29T11:15:38.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:45.039 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:45.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.212+0000 7efda2e46640 10 monclient: get_monmap_and_config 2024-03-29T11:15:45.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.212+0000 7efda2e46640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.212+0000 7efda2e46640 10 monclient: monmap: 2024-03-29T11:15:45.190 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:45.213635+0000 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:45.213635+0000 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:45.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.214+0000 7efda2e46640 1 Processor -- start 2024-03-29T11:15:45.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.215+0000 7efda2e46640 1 -- start start 2024-03-29T11:15:45.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.215+0000 7efda2e46640 10 monclient: init 2024-03-29T11:15:45.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:45.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c0a94a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-b con 0x7efd9c0a9880 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c149af0 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0aa820 con 0x7efd9c0a94a0 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c0afc10 con 0x7efd9c149af0 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c14a640 con 0x7efd9c0a9880 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.216+0000 7efda2e46640 10 monclient(hunting): authenticate will time out at 872.470145s 2024-03-29T11:15:45.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.219+0000 7efda1e44640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7efd9c0a94a0 0x7efd9c0af440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:48465/0 (socket says 172.21.15.3:48465) 2024-03-29T11:15:45.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.219+0000 7efda1e44640 1 -- 172.21.15.3:0/599967596 learned_addr learned my addr 172.21.15.3:0/599967596 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734470380 0 0) 0x7efd9c0aa820 con 0x7efd9c0a94a0 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:15:45.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.222+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd88003520 con 0x7efd9c0a94a0 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075870906 0 0) 0x7efd9c0afc10 con 0x7efd9c149af0 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:15:45.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.224+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd9c0aa820 con 0x7efd9c149af0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066778936 0 0) 0x7efd9c0aa820 con 0x7efd9c149af0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 >> v1:172.21.15.3:6790/0 conn(0x7efd9c0a9880 legacy=0x7efd9c0ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 >> v1:172.21.15.3:6789/0 conn(0x7efd9c0a94a0 legacy=0x7efd9c0af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 1 monclient: found mon.noname-c 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c072730 con 0x7efd9c149af0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.225+0000 7efda0e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda2e46640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:15:45.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda2e46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7efd940019f0 con 0x7efd9c149af0 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 10 monclient: dump: 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:45.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda2e46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:45.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 1 -- 172.21.15.3:0/599967596 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd94001c10 con 0x7efd9c149af0 2024-03-29T11:15:45.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda0e42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda2e46640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.226+0000 7efda2e46640 4 set_mon_vals no callback set 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.227+0000 7efda2e46640 10 monclient: shutdown 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.227+0000 7efda2e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.227+0000 7efda2e46640 1 -- 172.21.15.3:0/599967596 >> v1:172.21.15.32:6789/0 conn(0x7efd9c149af0 legacy=0x7efd9c149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.227+0000 7efda2e46640 1 -- 172.21.15.3:0/599967596 shutdown_connections 2024-03-29T11:15:45.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.227+0000 7efda2e46640 1 -- 172.21.15.3:0/599967596 >> 172.21.15.3:0/599967596 conn(0x7efd9c0b3820 msgr2=0x7efd9c0b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:45.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.231+0000 7efda2e46640 1 -- 172.21.15.3:0/599967596 shutdown_connections 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.231+0000 7efda2e46640 1 -- 172.21.15.3:0/599967596 wait complete. 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 10 monclient: build_initial_monmap 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 10 monclient: monmap: 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:45.209 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:45.210 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 librados: starting msgr at 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 librados: starting objecter 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 Processor -- start 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 -- start start 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 librados: setting wanted keys 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 1 librados: calling monclient init 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.232+0000 7efda2e46640 10 monclient: init 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.233+0000 7efda2e46640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:45.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.233+0000 7efda2e46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:45.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.233+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-c con 0x7efd9c0a94a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:45.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.237+0000 7efda1e44640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7efd9c0a94a0 0x7efd9c14ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:33961/0 (socket says 172.21.15.3:33961) 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.237+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-a con 0x7efd9c14b3e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 10 monclient(hunting): picked mon.noname-b con 0x7efd9c143860 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c143cf0 con 0x7efd9c14b3e0 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c14ba20 con 0x7efd9c143860 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd9c14beb0 con 0x7efd9c0a94a0 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda2e46640 10 monclient(hunting): authenticate will time out at 872.492385s 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efda1e44640 1 -- 172.21.15.3:0/1989760009 learned_addr learned my addr 172.21.15.3:0/1989760009 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 client.?.objecter ms_handle_connect 0x7efd9c0a94a0 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:45.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651919597 0 0) 0x7efd9c14beb0 con 0x7efd9c0a94a0 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.238+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd80004eb0 con 0x7efd9c0a94a0 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324734772 0 0) 0x7efd80004eb0 con 0x7efd9c0a94a0 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd80005300 con 0x7efd9c0a94a0 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7efd98001830 con 0x7efd9c0a94a0 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 10 monclient(hunting): dump: 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.239+0000 7efd927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.240+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3402201199 0 0) 0x7efd80005300 con 0x7efd9c0a94a0 2024-03-29T11:15:45.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.240+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 >> v1:172.21.15.32:6789/0 conn(0x7efd9c143860 legacy=0x7efd9c1db660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:45.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 >> v1:172.21.15.3:6789/0 conn(0x7efd9c14b3e0 legacy=0x7efd9c143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 1 monclient: found mon.c 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd9c152a80 con 0x7efd9c0a94a0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efd927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 10 monclient: _renew_subs 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd9c152c70 con 0x7efd9c0a94a0 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:45.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 10 monclient: _renew_subs 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd9c1dbf50 con 0x7efd9c0a94a0 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.242+0000 7efda2e46640 1 librados: init done 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd98001d60 con 0x7efd9c0a94a0 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7efd98001830 con 0x7efd9c0a94a0 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd927fc640 10 monclient: dump: 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:45.224 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efda2e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.244+0000 7efd93fff640 4 set_mon_vals no callback set 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.245+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7efd9801a850 con 0x7efd9c0a94a0 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.245+0000 7efda2e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.245+0000 7efda2e46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:45.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.245+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd9c1dc200 con 0x7efd9c0a94a0 2024-03-29T11:15:45.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.249+0000 7efda1643640 10 monclient: get_auth_request con 0x7efd8006dfb0 auth_method 0 2024-03-29T11:15:45.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 10 client.4262.objecter ms_handle_connect 0x7efd8006dfb0 2024-03-29T11:15:45.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7efd98054390 con 0x7efd9c0a94a0 2024-03-29T11:15:45.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 10 client.4262.objecter ms_dispatch 0x7efd9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:45.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:45.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:45.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.250+0000 7efd927fc640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:15:45.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.261+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd98050630 con 0x7efd9c0a94a0 2024-03-29T11:15:45.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.261+0000 7efd927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:45.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.261+0000 7efd927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:45.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.584+0000 7efda2e46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:45.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.584+0000 7efda2e46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:45.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.584+0000 7efda2e46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:15:45.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.584+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd9c06a400 con 0x7efd9c0a94a0 2024-03-29T11:15:45.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.585+0000 7efd927fc640 1 -- 172.21.15.3:0/1989760009 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7efd980507d0 con 0x7efd9c0a94a0 2024-03-29T11:15:45.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.585+0000 7efd927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:45.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.585+0000 7efd927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:45.565 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:45.565 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:45.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.591+0000 7efda2e46640 10 librados: watch_flush enter 2024-03-29T11:15:45.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.591+0000 7efda2e46640 10 librados: watch_flush exit 2024-03-29T11:15:45.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.591+0000 7efda2e46640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:15:45.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.591+0000 7efda2e46640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:15:45.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.592+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 >> v1:172.21.15.3:6816/844689376 conn(0x7efd8006dfb0 legacy=0x7efd80070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.592+0000 7efda2e46640 10 monclient: shutdown 2024-03-29T11:15:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.592+0000 7efda2e46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.592+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 >> v1:172.21.15.3:6790/0 conn(0x7efd9c0a94a0 legacy=0x7efd9c14ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:45.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.593+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 shutdown_connections 2024-03-29T11:15:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.593+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 >> 172.21.15.3:0/1989760009 conn(0x7efd9c0b3820 msgr2=0x7efd9c069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.593+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 shutdown_connections 2024-03-29T11:15:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.593+0000 7efda2e46640 1 -- 172.21.15.3:0/1989760009 wait complete. 2024-03-29T11:15:45.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:45.593+0000 7efda2e46640 1 librados: shutdown 2024-03-29T11:15:45.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:52.399 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:52.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.571+0000 7f04e1ae0640 10 monclient: get_monmap_and_config 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.571+0000 7f04e1ae0640 10 monclient: build_initial_monmap 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.571+0000 7f04e1ae0640 10 monclient: monmap: 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:15:52.572259+0000 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:15:52.572259+0000 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:52.549 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:52.550 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.572+0000 7f04e1ae0640 1 Processor -- start 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.572+0000 7f04e1ae0640 1 -- start start 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.572+0000 7f04e1ae0640 10 monclient: init 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-c con 0x7f04dc0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:52.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-b con 0x7f04dc0b4f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-a con 0x7f04dc0a9a90 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0aabd0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0aaeb0 con 0x7f04dc0b4b40 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc0ab190 con 0x7f04dc0b4f20 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04e1ae0640 10 monclient(hunting): authenticate will time out at 879.827302s 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04db7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f04dc0b4b40 0x7f04dc13be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:45044/0 (socket says 172.21.15.3:45044) 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.573+0000 7f04db7fe640 1 -- 172.21.15.3:0/3485476038 learned_addr learned my addr 172.21.15.3:0/3485476038 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645263749 0 0) 0x7f04dc0aaeb0 con 0x7f04dc0b4b40 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:52.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c4003520 con 0x7f04dc0b4b40 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563129046 0 0) 0x7f04dc0ab190 con 0x7f04dc0b4f20 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc0aaeb0 con 0x7f04dc0b4f20 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3256744676 0 0) 0x7f04c4003520 con 0x7f04dc0b4b40 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 >> v1:172.21.15.3:6790/0 conn(0x7f04dc0b4f20 legacy=0x7f04dc1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 >> v1:172.21.15.3:6789/0 conn(0x7f04dc0a9a90 legacy=0x7f04dc145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:52.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 monclient: found mon.noname-c 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0b45a0 con 0x7f04dc0b4b40 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f04cc001690 con 0x7f04dc0b4b40 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.574+0000 7f04da7fc640 10 monclient: dump: 2024-03-29T11:15:52.558 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04e1ae0640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04e1ae0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04cc0017f0 con 0x7f04dc0b4b40 2024-03-29T11:15:52.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 1 -- 172.21.15.3:0/3485476038 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f04cc001c00 con 0x7f04dc0b4b40 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04da7fc640 10 monclient: dump: 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:52.560 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04e1ae0640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.575+0000 7f04e1ae0640 4 set_mon_vals no callback set 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 10 monclient: shutdown 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 1 -- 172.21.15.3:0/3485476038 >> v1:172.21.15.32:6789/0 conn(0x7f04dc0b4b40 legacy=0x7f04dc13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 1 -- 172.21.15.3:0/3485476038 shutdown_connections 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 1 -- 172.21.15.3:0/3485476038 >> 172.21.15.3:0/3485476038 conn(0x7f04dc066a00 msgr2=0x7f04dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 1 -- 172.21.15.3:0/3485476038 shutdown_connections 2024-03-29T11:15:52.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.576+0000 7f04e1ae0640 1 -- 172.21.15.3:0/3485476038 wait complete. 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.577+0000 7f04e1ae0640 10 monclient: build_initial_monmap 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.577+0000 7f04e1ae0640 10 monclient: monmap: 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:15:52.571 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 librados: starting msgr at 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 librados: starting objecter 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 Processor -- start 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 -- start start 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 librados: setting wanted keys 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 1 librados: calling monclient init 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.578+0000 7f04e1ae0640 10 monclient: init 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-b con 0x7f04dc0a9a90 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-a con 0x7f04dc0b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient(hunting): picked mon.noname-c con 0x7f04dc144fb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:52.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc136930 con 0x7f04dc0b4b40 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc13e810 con 0x7f04dc0a9a90 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04dc13ec10 con 0x7f04dc144fb0 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04e1ae0640 10 monclient(hunting): authenticate will time out at 879.833170s 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04dbfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f04dc144fb0 0x7f04dc1e1d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58644/0 (socket says 172.21.15.3:58644) 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04dbfff640 1 -- 172.21.15.3:0/252508365 learned_addr learned my addr 172.21.15.3:0/252508365 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04c3fff640 10 client.?.objecter ms_handle_connect 0x7f04dc0a9a90 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04c3fff640 10 client.?.objecter ms_handle_connect 0x7f04dc144fb0 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.579+0000 7f04c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107155385 0 0) 0x7f04dc13e810 con 0x7f04dc0a9a90 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:52.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c8004eb0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 client.?.objecter ms_handle_connect 0x7f04dc0b4b40 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951972270 0 0) 0x7f04dc13ec10 con 0x7f04dc144fb0 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc13e810 con 0x7f04dc144fb0 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1905492847 0 0) 0x7f04c8004eb0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04dc13ec10 con 0x7f04dc0a9a90 2024-03-29T11:15:52.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f04cc001b00 con 0x7f04dc0a9a90 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): dump: 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.580+0000 7f04c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528372783 0 0) 0x7f04dc136930 con 0x7f04dc0b4b40 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c8004eb0 con 0x7f04dc0b4b40 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780689146 0 0) 0x7f04dc13ec10 con 0x7f04dc0a9a90 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 >> v1:172.21.15.3:6790/0 conn(0x7f04dc144fb0 legacy=0x7f04dc1e1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 >> v1:172.21.15.3:6789/0 conn(0x7f04dc0b4b40 legacy=0x7f04dc1d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 monclient: found mon.b 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc13eda0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 10 monclient: _renew_subs 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04dc13e020 con 0x7f04dc0a9a90 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 10 monclient: _renew_subs 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04dc13e550 con 0x7f04dc0a9a90 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.581+0000 7f04e1ae0640 1 librados: init done 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04cc004a90 con 0x7f04dc0a9a90 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:52.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f04cc001d80 con 0x7f04dc0a9a90 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04d97fa640 4 set_mon_vals no callback set 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.582+0000 7f04c3fff640 10 monclient: dump: 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:52.578 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.583+0000 7f04e1ae0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.583+0000 7f04e1ae0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f04cc01ac10 con 0x7f04dc0a9a90 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04e1ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04bc005080 con 0x7f04dc0a9a90 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f04cc0849f0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 10 client.4285.objecter ms_dispatch 0x7f04dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.584+0000 7f04c3fff640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.585+0000 7f04daffd640 10 monclient: get_auth_request con 0x7f04c806df30 auth_method 0 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.586+0000 7f04c3fff640 10 client.4285.objecter ms_handle_connect 0x7f04c806df30 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.592+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04cc0502d0 con 0x7f04dc0a9a90 2024-03-29T11:15:52.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.592+0000 7f04c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:52.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:52.592+0000 7f04c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:53.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.037+0000 7f04e1ae0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.037+0000 7f04e1ae0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.037+0000 7f04e1ae0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:15:53.014 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.037+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04bc005330 con 0x7f04dc0a9a90 2024-03-29T11:15:53.015 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:15:53.015 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:53.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.038+0000 7f04c3fff640 1 -- 172.21.15.3:0/252508365 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f04cc052200 con 0x7f04dc0a9a90 2024-03-29T11:15:53.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.038+0000 7f04c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:53.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.038+0000 7f04c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.041+0000 7f04e1ae0640 10 librados: watch_flush enter 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.041+0000 7f04e1ae0640 10 librados: watch_flush exit 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.041+0000 7f04e1ae0640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.041+0000 7f04e1ae0640 10 client.4285.objecter successfully canceled tick 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 >> v1:172.21.15.3:6816/844689376 conn(0x7f04c806df30 legacy=0x7f04c80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 10 monclient: shutdown 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 >> v1:172.21.15.32:6789/0 conn(0x7f04dc0a9a90 legacy=0x7f04dc1d7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 shutdown_connections 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 >> 172.21.15.3:0/252508365 conn(0x7f04dc066a00 msgr2=0x7f04dc0b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 shutdown_connections 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 -- 172.21.15.3:0/252508365 wait complete. 2024-03-29T11:15:53.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:15:53.042+0000 7f04e1ae0640 1 librados: shutdown 2024-03-29T11:15:53.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:59.843 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.007+0000 7fd2e8cdb640 10 monclient: get_monmap_and_config 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.007+0000 7fd2e8cdb640 10 monclient: build_initial_monmap 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.007+0000 7fd2e8cdb640 10 monclient: monmap: 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:00.008733+0000 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:00.008733+0000 2024-03-29T11:15:59.985 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.008+0000 7fd2e8cdb640 1 Processor -- start 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.009+0000 7fd2e8cdb640 1 -- start start 2024-03-29T11:15:59.986 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.009+0000 7fd2e8cdb640 10 monclient: init 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e40b2cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e40b3560 addr v1:172.21.15.32:6789/0 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e259c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd2e40b28d0 0x7fd2e413bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:46240/0 (socket says 172.21.15.3:46240) 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e259c640 1 -- 172.21.15.3:0/3504686867 learned_addr learned my addr 172.21.15.3:0/3504686867 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e40aab60 con 0x7fd2e40b2cb0 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e40aae40 con 0x7fd2e40b3560 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e40ab120 con 0x7fd2e40b28d0 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e8cdb640 10 monclient(hunting): authenticate will time out at 887.264651s 2024-03-29T11:15:59.988 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457439089 0 0) 0x7fd2e40ab120 con 0x7fd2e40b28d0 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.010+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2cc003520 con 0x7fd2e40b28d0 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.011+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1655351513 0 0) 0x7fd2cc003520 con 0x7fd2e40b28d0 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.011+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 >> v1:172.21.15.32:6789/0 conn(0x7fd2e40b3560 legacy=0x7fd2e41428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:59.989 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.011+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 >> v1:172.21.15.3:6789/0 conn(0x7fd2e40b2cb0 legacy=0x7fd2e4142190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 1 monclient: found mon.noname-b 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e413c380 con 0x7fd2e40b28d0 2024-03-29T11:15:59.991 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: _finish_auth 0 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd2d4001670 con 0x7fd2e40b28d0 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: dump: 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:59.992 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e8cdb640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e8cdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d40018d0 con 0x7fd2e40b28d0 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e8cdb640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e8cdb640 4 set_mon_vals no callback set 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 1 -- 172.21.15.3:0/3504686867 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd2d4001d20 con 0x7fd2e40b28d0 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.013+0000 7fd2e159a640 10 monclient: dump: 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:15:59.993 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.014+0000 7fd2e8cdb640 10 monclient: shutdown 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.014+0000 7fd2e8cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.014+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 >> v1:172.21.15.3:6790/0 conn(0x7fd2e40b28d0 legacy=0x7fd2e413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.014+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 shutdown_connections 2024-03-29T11:15:59.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.014+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 >> 172.21.15.3:0/3504686867 conn(0x7fd2e4066a00 msgr2=0x7fd2e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.015+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 shutdown_connections 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.015+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/3504686867 wait complete. 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.016+0000 7fd2e8cdb640 10 monclient: build_initial_monmap 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.016+0000 7fd2e8cdb640 10 monclient: monmap: 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:00.000 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 librados: starting msgr at 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 librados: starting objecter 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 Processor -- start 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 -- start start 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 librados: setting wanted keys 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 1 librados: calling monclient init 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.017+0000 7fd2e8cdb640 10 monclient: init 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-c con 0x7fd2e40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:00.001 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-a con 0x7fd2e4146040 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 10 monclient(hunting): picked mon.noname-b con 0x7fd2e41dff60 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e4132a50 con 0x7fd2e4146040 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e259c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd2e40b28d0 0x7fd2e41455c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:46800/0 (socket says 172.21.15.3:46800) 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e259c640 1 -- 172.21.15.3:0/1714990399 learned_addr learned my addr 172.21.15.3:0/1714990399 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.018+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e412eaa0 con 0x7fd2e41dff60 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fd2e40b28d0 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e412ecd0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 client.?.objecter ms_handle_connect 0x7fd2e41dff60 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2e8cdb640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2e8cdb640 10 monclient(hunting): authenticate will time out at 887.273230s 2024-03-29T11:16:00.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488467325 0 0) 0x7fd2e412ecd0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.019+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4004eb0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854926333 0 0) 0x7fd2e412eaa0 con 0x7fd2e41dff60 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e412ecd0 con 0x7fd2e41dff60 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3169394 0 0) 0x7fd2c4004eb0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2e412eaa0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd2d4001630 con 0x7fd2e40b28d0 2024-03-29T11:16:00.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): dump: 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:00.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3519398040 0 0) 0x7fd2e412eaa0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 >> v1:172.21.15.32:6789/0 conn(0x7fd2e41dff60 legacy=0x7fd2e41e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.020+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 >> v1:172.21.15.3:6789/0 conn(0x7fd2e4146040 legacy=0x7fd2e412f6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 1 monclient: found mon.c 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e412efb0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:16:00.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 10 monclient: _renew_subs 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e4144750 con 0x7fd2e40b28d0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 10 monclient: _renew_subs 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e412de30 con 0x7fd2e40b28d0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.021+0000 7fd2e8cdb640 1 librados: init done 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2d4004c70 con 0x7fd2e40b28d0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd2d4004f80 con 0x7fd2e40b28d0 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2d2ffd640 10 monclient: dump: 2024-03-29T11:16:00.006 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.022+0000 7fd2e0d99640 4 set_mon_vals no callback set 2024-03-29T11:16:00.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.023+0000 7fd2e8cdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.024+0000 7fd2e8cdb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.024+0000 7fd2e8cdb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.024+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c8005080 con 0x7fd2e40b28d0 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.024+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fd2d401af50 con 0x7fd2e40b28d0 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fd2d4084d00 con 0x7fd2e40b28d0 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2d2ffd640 10 client.4274.objecter ms_dispatch 0x7fd2e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:00.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2d2ffd640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2d2ffd640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2e1d9b640 10 monclient: get_auth_request con 0x7fd2c406dd70 auth_method 0 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.025+0000 7fd2d2ffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.026+0000 7fd2d2ffd640 10 client.4274.objecter ms_handle_connect 0x7fd2c406dd70 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.031+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2d40505e0 con 0x7fd2e40b28d0 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.031+0000 7fd2d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:00.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.031+0000 7fd2d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:00.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.343+0000 7fd2e8cdb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.343+0000 7fd2e8cdb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.343+0000 7fd2e8cdb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:00.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.343+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2c8005330 con 0x7fd2e40b28d0 2024-03-29T11:16:00.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.344+0000 7fd2d2ffd640 1 -- 172.21.15.3:0/1714990399 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fd2d4052510 con 0x7fd2e40b28d0 2024-03-29T11:16:00.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.344+0000 7fd2d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:00.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.344+0000 7fd2d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:00.323 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:00.323 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:00.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 10 librados: watch_flush enter 2024-03-29T11:16:00.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 10 librados: watch_flush exit 2024-03-29T11:16:00.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:16:00.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:16:00.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 >> v1:172.21.15.3:6816/844689376 conn(0x7fd2c406dd70 legacy=0x7fd2c4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 10 monclient: shutdown 2024-03-29T11:16:00.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:00.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 >> v1:172.21.15.3:6790/0 conn(0x7fd2e40b28d0 legacy=0x7fd2e41455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:00.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.348+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 shutdown_connections 2024-03-29T11:16:00.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.349+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 >> 172.21.15.3:0/1714990399 conn(0x7fd2e4066a00 msgr2=0x7fd2e4135600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:00.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.349+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 shutdown_connections 2024-03-29T11:16:00.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.349+0000 7fd2e8cdb640 1 -- 172.21.15.3:0/1714990399 wait complete. 2024-03-29T11:16:00.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:00.349+0000 7fd2e8cdb640 1 librados: shutdown 2024-03-29T11:16:00.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:07.148 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.322+0000 7fc340081640 10 monclient: get_monmap_and_config 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.322+0000 7fc340081640 10 monclient: build_initial_monmap 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.322+0000 7fc340081640 10 monclient: monmap: 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:07.323625+0000 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:07.323625+0000 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.300 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:07.301 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:07.301 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:07.301 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.324+0000 7fc340081640 1 Processor -- start 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.324+0000 7fc340081640 1 -- start start 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.324+0000 7fc340081640 10 monclient: init 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient(hunting): picked mon.noname-c con 0x7fc33806e4f0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient(hunting): picked mon.noname-b con 0x7fc338070e90 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient(hunting): picked mon.noname-a con 0x7fc33813ba90 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33813e300 con 0x7fc33813ba90 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338070790 con 0x7fc33806e4f0 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338070a70 con 0x7fc338070e90 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:07.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.325+0000 7fc340081640 10 monclient(hunting): authenticate will time out at 894.579382s 2024-03-29T11:16:07.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33d61c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fc338070e90 0x7fc33813b380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:45743/0 (socket says 172.21.15.3:45743) 2024-03-29T11:16:07.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33d61c640 1 -- 172.21.15.3:0/3099155663 learned_addr learned my addr 172.21.15.3:0/3099155663 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:07.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517002486 0 0) 0x7fc338070a70 con 0x7fc338070e90 2024-03-29T11:16:07.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 10 monclient(hunting): _negotiate 2024-03-29T11:16:07.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:07.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:07.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:16:07.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.331+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc324003520 con 0x7fc338070e90 2024-03-29T11:16:07.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1300231616 0 0) 0x7fc324003520 con 0x7fc338070e90 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 >> v1:172.21.15.32:6789/0 conn(0x7fc33806e4f0 legacy=0x7fc33813e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 >> v1:172.21.15.3:6789/0 conn(0x7fc33813ba90 legacy=0x7fc3381499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: _finish_hunting 0 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 monclient: found mon.noname-b 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33806f470 con 0x7fc338070e90 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: _finish_auth 0 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: _check_auth_tickets 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc328001670 con 0x7fc338070e90 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: dump: 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc340081640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:16:07.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc340081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc3280018d0 con 0x7fc338070e90 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 1 -- 172.21.15.3:0/3099155663 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc328001ca0 con 0x7fc338070e90 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc33ce1b640 10 monclient: dump: 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:07.312 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc340081640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.332+0000 7fc340081640 4 set_mon_vals no callback set 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.333+0000 7fc340081640 10 monclient: shutdown 2024-03-29T11:16:07.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.333+0000 7fc340081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:07.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.333+0000 7fc340081640 1 -- 172.21.15.3:0/3099155663 >> v1:172.21.15.3:6790/0 conn(0x7fc338070e90 legacy=0x7fc33813b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.333+0000 7fc340081640 1 -- 172.21.15.3:0/3099155663 shutdown_connections 2024-03-29T11:16:07.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.333+0000 7fc340081640 1 -- 172.21.15.3:0/3099155663 >> 172.21.15.3:0/3099155663 conn(0x7fc3380b3c10 msgr2=0x7fc3380b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:07.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.337+0000 7fc340081640 1 -- 172.21.15.3:0/3099155663 shutdown_connections 2024-03-29T11:16:07.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.337+0000 7fc340081640 1 -- 172.21.15.3:0/3099155663 wait complete. 2024-03-29T11:16:07.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.338+0000 7fc340081640 10 monclient: build_initial_monmap 2024-03-29T11:16:07.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.338+0000 7fc340081640 10 monclient: monmap: 2024-03-29T11:16:07.315 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:07.315 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:07.315 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:07.316 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 librados: starting msgr at 2024-03-29T11:16:07.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 librados: starting objecter 2024-03-29T11:16:07.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 Processor -- start 2024-03-29T11:16:07.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 -- start start 2024-03-29T11:16:07.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 librados: setting wanted keys 2024-03-29T11:16:07.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 1 librados: calling monclient init 2024-03-29T11:16:07.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.339+0000 7fc340081640 10 monclient: init 2024-03-29T11:16:07.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.340+0000 7fc340081640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:07.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.340+0000 7fc340081640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:07.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.340+0000 7fc340081640 10 monclient(hunting): picked mon.noname-c con 0x7fc33806e4f0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.343+0000 7fc340081640 10 monclient(hunting): picked mon.noname-a con 0x7fc33813ba90 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 10 monclient(hunting): picked mon.noname-b con 0x7fc33813c4e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3381cb890 con 0x7fc33813ba90 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3381d5940 con 0x7fc33813c4e0 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3381d5ad0 con 0x7fc33806e4f0 2024-03-29T11:16:07.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:07.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.345+0000 7fc340081640 10 monclient(hunting): authenticate will time out at 894.599399s 2024-03-29T11:16:07.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.348+0000 7fc33de1d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fc33806e4f0 0x7fc3381d5230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:45857/0 (socket says 172.21.15.3:45857) 2024-03-29T11:16:07.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.348+0000 7fc33de1d640 1 -- 172.21.15.3:0/913698439 learned_addr learned my addr 172.21.15.3:0/913698439 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:07.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.348+0000 7fc32e7fc640 10 client.?.objecter ms_handle_connect 0x7fc33806e4f0 2024-03-29T11:16:07.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.348+0000 7fc32e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:07.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238723380 0 0) 0x7fc3381d5ad0 con 0x7fc33806e4f0 2024-03-29T11:16:07.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:07.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:07.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:07.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:16:07.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc31c004eb0 con 0x7fc33806e4f0 2024-03-29T11:16:07.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846252048 0 0) 0x7fc31c004eb0 con 0x7fc33806e4f0 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3381d5ad0 con 0x7fc33806e4f0 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc334001960 con 0x7fc33806e4f0 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:07.329 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3450593309 0 0) 0x7fc3381d5ad0 con 0x7fc33806e4f0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 >> v1:172.21.15.32:6789/0 conn(0x7fc33813c4e0 legacy=0x7fc3381e0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.349+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 >> v1:172.21.15.3:6789/0 conn(0x7fc33813ba90 legacy=0x7fc338136d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 1 monclient: found mon.c 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33813dbf0 con 0x7fc33806e4f0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:07.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc334001c80 con 0x7fc33806e4f0 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc334003a40 con 0x7fc33806e4f0 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32e7fc640 10 monclient: dump: 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:07.331 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.350+0000 7fc32ffff640 4 set_mon_vals no callback set 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 5 monclient: authenticate success, global_id 4280 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 10 monclient: _renew_subs 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33813de40 con 0x7fc33806e4f0 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 10 monclient: _renew_subs 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.351+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3380ac260 con 0x7fc33806e4f0 2024-03-29T11:16:07.332 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.352+0000 7fc340081640 1 librados: init done 2024-03-29T11:16:07.333 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.353+0000 7fc340081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.353+0000 7fc340081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.353+0000 7fc340081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.353+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33813c1d0 con 0x7fc33806e4f0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.355+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fc334003ba0 con 0x7fc33806e4f0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.355+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fc334084af0 con 0x7fc33806e4f0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc33d61c640 10 monclient: get_auth_request con 0x7fc31c06dce0 auth_method 0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc32e7fc640 10 client.4280.objecter ms_dispatch 0x7fc338000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc32e7fc640 3 client.4280.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc32e7fc640 3 client.4280.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc32e7fc640 20 client.4280.objecter dump_active .. 0 homeless 2024-03-29T11:16:07.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.356+0000 7fc32e7fc640 10 client.4280.objecter ms_handle_connect 0x7fc31c06dce0 2024-03-29T11:16:07.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.365+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc3340503d0 con 0x7fc33806e4f0 2024-03-29T11:16:07.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.365+0000 7fc32e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:07.342 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.365+0000 7fc32e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:07.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.700+0000 7fc340081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:07.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.700+0000 7fc340081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:07.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.700+0000 7fc340081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:07.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.700+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc33806a400 con 0x7fc33806e4f0 2024-03-29T11:16:07.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.702+0000 7fc32e7fc640 1 -- 172.21.15.3:0/913698439 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fc334052300 con 0x7fc33806e4f0 2024-03-29T11:16:07.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.702+0000 7fc32e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:07.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.702+0000 7fc32e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:07.680 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:07.680 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:07.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 10 librados: watch_flush enter 2024-03-29T11:16:07.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 10 librados: watch_flush exit 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 20 client.4280.objecter shutdown clearing up homeless session... 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 10 client.4280.objecter successfully canceled tick 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 >> v1:172.21.15.3:6816/844689376 conn(0x7fc31c06dce0 legacy=0x7fc31c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 10 monclient: shutdown 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 >> v1:172.21.15.3:6790/0 conn(0x7fc33806e4f0 legacy=0x7fc3381d5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 shutdown_connections 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.706+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 >> 172.21.15.3:0/913698439 conn(0x7fc3380b3c10 msgr2=0x7fc3381cbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.707+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 shutdown_connections 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.707+0000 7fc340081640 1 -- 172.21.15.3:0/913698439 wait complete. 2024-03-29T11:16:07.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:07.707+0000 7fc340081640 1 librados: shutdown 2024-03-29T11:16:07.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:14.510 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.686+0000 7f2199608640 10 monclient: get_monmap_and_config 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.686+0000 7f2199608640 10 monclient: build_initial_monmap 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.686+0000 7f2199608640 10 monclient: monmap: 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:14.687446+0000 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:14.687446+0000 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:14.664 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.688+0000 7f2199608640 1 Processor -- start 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.688+0000 7f2199608640 1 -- start start 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.688+0000 7f2199608640 10 monclient: init 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.688+0000 7f2199608640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient(hunting): picked mon.noname-c con 0x7f21940af460 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient(hunting): picked mon.noname-a con 0x7f21941499a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:14.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient(hunting): picked mon.noname-b con 0x7f2194143df0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:14.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219414a050 con 0x7f21941499a0 2024-03-29T11:16:14.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21940aef20 con 0x7f21940af460 2024-03-29T11:16:14.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2194146bb0 con 0x7f2194143df0 2024-03-29T11:16:14.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:14.667 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.689+0000 7f2199608640 10 monclient(hunting): authenticate will time out at 901.943116s 2024-03-29T11:16:14.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.690+0000 7f2193fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f21940af460 0x7f21941464a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:53750/0 (socket says 172.21.15.3:53750) 2024-03-29T11:16:14.668 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.690+0000 7f2193fff640 1 -- 172.21.15.3:0/1300645324 learned_addr learned my addr 172.21.15.3:0/1300645324 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.692+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295378045 0 0) 0x7f21940aef20 con 0x7f21940af460 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.693+0000 7f2192ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.693+0000 7f2192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.693+0000 7f2192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.693+0000 7f2192ffd640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:16:14.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.693+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f217c003520 con 0x7f21940af460 2024-03-29T11:16:14.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281162227 0 0) 0x7f2194146bb0 con 0x7f2194143df0 2024-03-29T11:16:14.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:14.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:14.671 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:14.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:16:14.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.694+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21940aef20 con 0x7f2194143df0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297593455 0 0) 0x7f21940aef20 con 0x7f2194143df0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 >> v1:172.21.15.32:6789/0 conn(0x7f21940af460 legacy=0x7f21941464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 >> v1:172.21.15.3:6789/0 conn(0x7f21941499a0 legacy=0x7f21941436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 1 monclient: found mon.noname-b 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.695+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219414a4e0 con 0x7f2194143df0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:14.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f218c0019f0 con 0x7f2194143df0 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: dump: 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:14.674 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f218c001c10 con 0x7f2194143df0 2024-03-29T11:16:14.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:14.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 1 -- 172.21.15.3:0/1300645324 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f218c001fa0 con 0x7f2194143df0 2024-03-29T11:16:14.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2192ffd640 10 monclient: dump: 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2199608640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:16:14.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2199608640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.696+0000 7f2199608640 4 set_mon_vals no callback set 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.697+0000 7f2199608640 10 monclient: shutdown 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.697+0000 7f2199608640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.697+0000 7f2199608640 1 -- 172.21.15.3:0/1300645324 >> v1:172.21.15.3:6790/0 conn(0x7f2194143df0 legacy=0x7f2194151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.697+0000 7f2199608640 1 -- 172.21.15.3:0/1300645324 shutdown_connections 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.698+0000 7f2199608640 1 -- 172.21.15.3:0/1300645324 >> 172.21.15.3:0/1300645324 conn(0x7f2194066a00 msgr2=0x7f2194066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.698+0000 7f2199608640 1 -- 172.21.15.3:0/1300645324 shutdown_connections 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.698+0000 7f2199608640 1 -- 172.21.15.3:0/1300645324 wait complete. 2024-03-29T11:16:14.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.699+0000 7f2199608640 10 monclient: build_initial_monmap 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.699+0000 7f2199608640 10 monclient: monmap: 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.699+0000 7f2199608640 1 librados: starting msgr at 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.699+0000 7f2199608640 1 librados: starting objecter 2024-03-29T11:16:14.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.700+0000 7f2199608640 1 Processor -- start 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.700+0000 7f2199608640 1 -- start start 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.700+0000 7f2199608640 1 librados: setting wanted keys 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.700+0000 7f2199608640 1 librados: calling monclient init 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.700+0000 7f2199608640 10 monclient: init 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:14.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient(hunting): picked mon.noname-a con 0x7f21940af460 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient(hunting): picked mon.noname-b con 0x7f2194143df0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient(hunting): picked mon.noname-c con 0x7f21941499a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21940ab710 con 0x7f21940af460 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21941e7ec0 con 0x7f2194143df0 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21941e7bc0 con 0x7f21941499a0 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.701+0000 7f2199608640 10 monclient(hunting): authenticate will time out at 901.955121s 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.703+0000 7f21937fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2194143df0 0x7f21941daba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:59182/0 (socket says 172.21.15.3:59182) 2024-03-29T11:16:14.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.703+0000 7f21937fe640 1 -- 172.21.15.3:0/1173449838 learned_addr learned my addr 172.21.15.3:0/1173449838 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 client.?.objecter ms_handle_connect 0x7f2194143df0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499185677 0 0) 0x7f21941e7ec0 con 0x7f2194143df0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2180004eb0 con 0x7f2194143df0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1385018308 0 0) 0x7f2180004eb0 con 0x7f2194143df0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21941e7ec0 con 0x7f2194143df0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.704+0000 7f2190ff9640 10 client.?.objecter ms_handle_connect 0x7f21941499a0 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:14.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f2184001990 con 0x7f2194143df0 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600770149 0 0) 0x7f21941e7bc0 con 0x7f21941499a0 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2180004eb0 con 0x7f21941499a0 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 client.?.objecter ms_handle_connect 0x7f21940af460 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161081613 0 0) 0x7f21940ab710 con 0x7f21940af460 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:14.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21941e7bc0 con 0x7f21940af460 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3125545933 0 0) 0x7f2180004eb0 con 0x7f21941499a0 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21940ab710 con 0x7f21941499a0 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f218c001930 con 0x7f21941499a0 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.705+0000 7f2190ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.688 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 711588536 0 0) 0x7f21941e7bc0 con 0x7f21940af460 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2180004eb0 con 0x7f21940af460 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296765438 0 0) 0x7f21940ab710 con 0x7f21941499a0 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 >> v1:172.21.15.32:6789/0 conn(0x7f2194143df0 legacy=0x7f21941daba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 >> v1:172.21.15.3:6789/0 conn(0x7f21940af460 legacy=0x7f21941e74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 monclient: found mon.c 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:14.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21940ae570 con 0x7f21941499a0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2190ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2199608640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2199608640 10 monclient: _renew_subs 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2199608640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.706+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2194071980 con 0x7f21941499a0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2199608640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2199608640 10 monclient: _renew_subs 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2199608640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21941db490 con 0x7f21941499a0 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2199608640 1 librados: init done 2024-03-29T11:16:14.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f218c0013a0 con 0x7f21941499a0 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f218c001590 con 0x7f21941499a0 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f2190ff9640 10 monclient: dump: 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:14.691 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:14.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.707+0000 7f21927fc640 4 set_mon_vals no callback set 2024-03-29T11:16:14.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.708+0000 7f2199608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:14.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.708+0000 7f2199608640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:14.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.709+0000 7f2199608640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:14.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.709+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2174005080 con 0x7f21941499a0 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.711+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f218c019c30 con 0x7f21941499a0 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.711+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f218c083a20 con 0x7f21941499a0 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.711+0000 7f2190ff9640 10 client.4286.objecter ms_dispatch 0x7f2194000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.712+0000 7f2190ff9640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.712+0000 7f2190ff9640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.712+0000 7f2190ff9640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.712+0000 7f2193fff640 10 monclient: get_auth_request con 0x7f218006e130 auth_method 0 2024-03-29T11:16:14.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.713+0000 7f2190ff9640 10 client.4286.objecter ms_handle_connect 0x7f218006e130 2024-03-29T11:16:14.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.718+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f218c04f300 con 0x7f21941499a0 2024-03-29T11:16:14.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.718+0000 7f2190ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:14.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:14.718+0000 7f2190ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:15.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.062+0000 7f2199608640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.062+0000 7f2199608640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.062+0000 7f2199608640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:15.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.062+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2174005330 con 0x7f21941499a0 2024-03-29T11:16:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.063+0000 7f2190ff9640 1 -- 172.21.15.3:0/1173449838 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f218c051230 con 0x7f21941499a0 2024-03-29T11:16:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.063+0000 7f2190ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:15.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.063+0000 7f2190ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:15.040 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:15.041 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:15.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 10 librados: watch_flush enter 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 10 librados: watch_flush exit 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 10 client.4286.objecter successfully canceled tick 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 >> v1:172.21.15.3:6816/844689376 conn(0x7f218006e130 legacy=0x7f21800705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 10 monclient: shutdown 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:15.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.066+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 >> v1:172.21.15.3:6790/0 conn(0x7f21941499a0 legacy=0x7f21940a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:15.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.069+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 shutdown_connections 2024-03-29T11:16:15.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.069+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 >> 172.21.15.3:0/1173449838 conn(0x7f2194066a00 msgr2=0x7f21941434b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:15.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.070+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 shutdown_connections 2024-03-29T11:16:15.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.070+0000 7f2199608640 1 -- 172.21.15.3:0/1173449838 wait complete. 2024-03-29T11:16:15.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:15.071+0000 7f2199608640 1 librados: shutdown 2024-03-29T11:16:15.065 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:21.872 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:22.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.048+0000 7f4137900640 10 monclient: get_monmap_and_config 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.048+0000 7f4137900640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.048+0000 7f4137900640 10 monclient: monmap: 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:22.049236+0000 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:22.049236+0000 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:22.026 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.049+0000 7f4137900640 1 Processor -- start 2024-03-29T11:16:22.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.049+0000 7f4137900640 1 -- start start 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.049+0000 7f4137900640 10 monclient: init 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient(hunting): picked mon.noname-a con 0x7f41300a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient(hunting): picked mon.noname-c con 0x7f41300aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient(hunting): picked mon.noname-b con 0x7f4130146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300aed20 con 0x7f41300a98a0 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41300aa8b0 con 0x7f41300aab00 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4130146a60 con 0x7f4130146f80 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.050+0000 7f4137900640 10 monclient(hunting): authenticate will time out at 909.304729s 2024-03-29T11:16:22.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.055+0000 7f4135e9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4130146f80 0x7f4130151cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:49263/0 (socket says 172.21.15.3:49263) 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.055+0000 7f4135e9d640 1 -- 172.21.15.3:0/850074292 learned_addr learned my addr 172.21.15.3:0/850074292 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456790626 0 0) 0x7f4130146a60 con 0x7f4130146f80 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:16:22.034 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c003520 con 0x7f4130146f80 2024-03-29T11:16:22.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1555998985 0 0) 0x7f411c003520 con 0x7f4130146f80 2024-03-29T11:16:22.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 >> v1:172.21.15.32:6789/0 conn(0x7f41300aab00 legacy=0x7f41301437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.057+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 >> v1:172.21.15.3:6789/0 conn(0x7f41300a98a0 legacy=0x7f41300ae610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 1 monclient: found mon.noname-b 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4130143ef0 con 0x7f4130146f80 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4120001690 con 0x7f4130146f80 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:22.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: dump: 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4120001860 con 0x7f4130146f80 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 1 -- 172.21.15.3:0/850074292 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4120001c70 con 0x7f4130146f80 2024-03-29T11:16:22.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.058+0000 7f4127fff640 10 monclient: dump: 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:22.038 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 4 set_mon_vals no callback set 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 10 monclient: shutdown 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.060+0000 7f4137900640 1 -- 172.21.15.3:0/850074292 >> v1:172.21.15.3:6790/0 conn(0x7f4130146f80 legacy=0x7f4130151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.061+0000 7f4137900640 1 -- 172.21.15.3:0/850074292 shutdown_connections 2024-03-29T11:16:22.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.061+0000 7f4137900640 1 -- 172.21.15.3:0/850074292 >> 172.21.15.3:0/850074292 conn(0x7f41300b3c10 msgr2=0x7f41300b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.061+0000 7f4137900640 1 -- 172.21.15.3:0/850074292 shutdown_connections 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.061+0000 7f4137900640 1 -- 172.21.15.3:0/850074292 wait complete. 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 10 monclient: build_initial_monmap 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 10 monclient: monmap: 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:22.040 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 librados: starting msgr at 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 librados: starting objecter 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 Processor -- start 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 -- start start 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 librados: setting wanted keys 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 1 librados: calling monclient init 2024-03-29T11:16:22.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.062+0000 7f4137900640 10 monclient: init 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient(hunting): picked mon.noname-c con 0x7f41300a98a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient(hunting): picked mon.noname-b con 0x7f41300aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient(hunting): picked mon.noname-a con 0x7f4130146f80 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:22.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4130141250 con 0x7f4130146f80 2024-03-29T11:16:22.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413011c1f0 con 0x7f41300aab00 2024-03-29T11:16:22.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f413013dba0 con 0x7f41300a98a0 2024-03-29T11:16:22.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:22.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.063+0000 7f4137900640 10 monclient(hunting): authenticate will time out at 909.317814s 2024-03-29T11:16:22.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4134e9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f41300aab00 0x7f41301416e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:57281/0 (socket says 172.21.15.3:57281) 2024-03-29T11:16:22.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4134e9b640 1 -- 172.21.15.3:0/3900417200 learned_addr learned my addr 172.21.15.3:0/3900417200 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:22.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4125ffb640 10 client.?.objecter ms_handle_connect 0x7f41300aab00 2024-03-29T11:16:22.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4125ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4125ffb640 10 client.?.objecter ms_handle_connect 0x7f4130146f80 2024-03-29T11:16:22.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.070+0000 7f4125ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596503483 0 0) 0x7f4130141250 con 0x7f4130146f80 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114004eb0 con 0x7f4130146f80 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 client.?.objecter ms_handle_connect 0x7f41300a98a0 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.071+0000 7f4125ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896468400 0 0) 0x7f413013dba0 con 0x7f41300a98a0 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:22.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114006230 con 0x7f41300a98a0 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2382601722 0 0) 0x7f4114006230 con 0x7f41300a98a0 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41140066a0 con 0x7f41300a98a0 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f412c001510 con 0x7f41300a98a0 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.053 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.072+0000 7f4125ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 433058514 0 0) 0x7f41140066a0 con 0x7f41300a98a0 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 >> v1:172.21.15.32:6789/0 conn(0x7f41300aab00 legacy=0x7f41301416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 >> v1:172.21.15.3:6789/0 conn(0x7f4130146f80 legacy=0x7f41302002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 1 monclient: found mon.c 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:22.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413013e940 con 0x7f41300a98a0 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.073+0000 7f4125ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:16:22.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 10 monclient: _renew_subs 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41300705e0 con 0x7f41300a98a0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 10 monclient: _renew_subs 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4130141fd0 con 0x7f41300a98a0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4137900640 1 librados: init done 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f412c0019c0 con 0x7f41300a98a0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f412c001510 con 0x7f41300a98a0 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f4125ffb640 10 monclient: dump: 2024-03-29T11:16:22.056 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.074+0000 7f41277fe640 4 set_mon_vals no callback set 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.076+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f412c001c30 con 0x7f41300a98a0 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.077+0000 7f4137900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.077+0000 7f4137900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.077+0000 7f4137900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.077+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110005080 con 0x7f41300a98a0 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.077+0000 7f4134e9b640 10 monclient: get_auth_request con 0x7f411406e570 auth_method 0 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f412c001330 con 0x7f41300a98a0 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 10 client.4292.objecter ms_dispatch 0x7f4130000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:16:22.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.078+0000 7f4125ffb640 10 client.4292.objecter ms_handle_connect 0x7f411406e570 2024-03-29T11:16:22.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.089+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f412c04f310 con 0x7f41300a98a0 2024-03-29T11:16:22.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.089+0000 7f4125ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:22.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.089+0000 7f4125ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:22.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.425+0000 7f4137900640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:22.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.425+0000 7f4137900640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:22.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.425+0000 7f4137900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:22.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.425+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4110005330 con 0x7f41300a98a0 2024-03-29T11:16:22.406 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:22.406 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:22.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.427+0000 7f4125ffb640 1 -- 172.21.15.3:0/3900417200 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f412c04f4b0 con 0x7f41300a98a0 2024-03-29T11:16:22.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.427+0000 7f4125ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:22.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.427+0000 7f4125ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 10 librados: watch_flush enter 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 10 librados: watch_flush exit 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 >> v1:172.21.15.3:6816/844689376 conn(0x7f411406e570 legacy=0x7f4114070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 10 monclient: shutdown 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 >> v1:172.21.15.3:6790/0 conn(0x7f41300a98a0 legacy=0x7f413013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 shutdown_connections 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.431+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 >> 172.21.15.3:0/3900417200 conn(0x7f41300b3c10 msgr2=0x7f41301435b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.432+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 shutdown_connections 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.432+0000 7f4137900640 1 -- 172.21.15.3:0/3900417200 wait complete. 2024-03-29T11:16:22.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:22.432+0000 7f4137900640 1 librados: shutdown 2024-03-29T11:16:22.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:29.235 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.406+0000 7f1795081640 10 monclient: get_monmap_and_config 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.406+0000 7f1795081640 10 monclient: build_initial_monmap 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.406+0000 7f1795081640 10 monclient: monmap: 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:29.407680+0000 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:29.407680+0000 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:29.384 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:29.385 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:29.385 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.409+0000 7f1795081640 1 Processor -- start 2024-03-29T11:16:29.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.409+0000 7f1795081640 1 -- start start 2024-03-29T11:16:29.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.409+0000 7f1795081640 10 monclient: init 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient(hunting): picked mon.noname-c con 0x7f17900b1f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient(hunting): picked mon.noname-b con 0x7f1790146400 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient(hunting): picked mon.noname-a con 0x7f179014fd10 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179006ed20 con 0x7f179014fd10 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179006ef90 con 0x7f17900b1f20 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900b42e0 con 0x7f1790146400 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:29.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.410+0000 7f1795081640 10 monclient(hunting): authenticate will time out at 916.664600s 2024-03-29T11:16:29.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.417+0000 7f178f59e640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f179014fd10 0x7f1790152100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53557/0 (socket says 172.21.15.3:53557) 2024-03-29T11:16:29.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.417+0000 7f178f59e640 1 -- 172.21.15.3:0/882652207 learned_addr learned my addr 172.21.15.3:0/882652207 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017309713 0 0) 0x7f17900b42e0 con 0x7f1790146400 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1778003520 con 0x7f1790146400 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496503295 0 0) 0x7f179006ed20 con 0x7f179014fd10 2024-03-29T11:16:29.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.418+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17900b42e0 con 0x7f179014fd10 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 346925917 0 0) 0x7f1778003520 con 0x7f1790146400 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 >> v1:172.21.15.32:6789/0 conn(0x7f17900b1f20 legacy=0x7f17901434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 >> v1:172.21.15.3:6789/0 conn(0x7f179014fd10 legacy=0x7f1790152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 monclient: found mon.noname-b 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17900b3690 con 0x7f1790146400 2024-03-29T11:16:29.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f1795081640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f1795081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1784001690 con 0x7f1790146400 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: dump: 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.399 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f1795081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:29.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 1 -- 172.21.15.3:0/882652207 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1784001870 con 0x7f1790146400 2024-03-29T11:16:29.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f178dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:29.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f1795081640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.419+0000 7f1795081640 4 set_mon_vals no callback set 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.420+0000 7f1795081640 10 monclient: shutdown 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.420+0000 7f1795081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.420+0000 7f1795081640 1 -- 172.21.15.3:0/882652207 >> v1:172.21.15.3:6790/0 conn(0x7f1790146400 legacy=0x7f1790146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.421+0000 7f1795081640 1 -- 172.21.15.3:0/882652207 shutdown_connections 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.421+0000 7f1795081640 1 -- 172.21.15.3:0/882652207 >> 172.21.15.3:0/882652207 conn(0x7f1790066a00 msgr2=0x7f1790066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.421+0000 7f1795081640 1 -- 172.21.15.3:0/882652207 shutdown_connections 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.421+0000 7f1795081640 1 -- 172.21.15.3:0/882652207 wait complete. 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.422+0000 7f1795081640 10 monclient: build_initial_monmap 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.422+0000 7f1795081640 10 monclient: monmap: 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:29.403 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.422+0000 7f1795081640 1 librados: starting msgr at 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.422+0000 7f1795081640 1 librados: starting objecter 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.423+0000 7f1795081640 1 Processor -- start 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.423+0000 7f1795081640 1 -- start start 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.423+0000 7f1795081640 1 librados: setting wanted keys 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.423+0000 7f1795081640 1 librados: calling monclient init 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.423+0000 7f1795081640 10 monclient: init 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:29.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient(hunting): picked mon.noname-a con 0x7f17900b1f20 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient(hunting): picked mon.noname-b con 0x7f1790146400 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient(hunting): picked mon.noname-c con 0x7f179014fd10 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179014adf0 con 0x7f17900b1f20 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17900b7350 con 0x7f1790146400 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f179013d7e0 con 0x7f179014fd10 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:29.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.424+0000 7f1795081640 10 monclient(hunting): authenticate will time out at 916.678322s 2024-03-29T11:16:29.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.429+0000 7f178f59e640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f179014fd10 0x7f17901de930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42837/0 (socket says 172.21.15.3:42837) 2024-03-29T11:16:29.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.429+0000 7f178f59e640 1 -- 172.21.15.3:0/1600265137 learned_addr learned my addr 172.21.15.3:0/1600265137 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:29.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.429+0000 7f17777fe640 10 client.?.objecter ms_handle_connect 0x7f179014fd10 2024-03-29T11:16:29.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.429+0000 7f17777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:29.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.430+0000 7f17777fe640 10 client.?.objecter ms_handle_connect 0x7f17900b1f20 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.431+0000 7f17777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.431+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845606716 0 0) 0x7f179013d7e0 con 0x7f179014fd10 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f177c004eb0 con 0x7f179014fd10 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199266252 0 0) 0x7f179014adf0 con 0x7f17900b1f20 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:29.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179013d7e0 con 0x7f17900b1f20 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 client.?.objecter ms_handle_connect 0x7f1790146400 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.432+0000 7f17777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840346934 0 0) 0x7f17900b7350 con 0x7f1790146400 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.433+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f179014adf0 con 0x7f1790146400 2024-03-29T11:16:29.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.435+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2710593790 0 0) 0x7f179014adf0 con 0x7f1790146400 2024-03-29T11:16:29.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17900b7350 con 0x7f1790146400 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1784001c00 con 0x7f1790146400 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): dump: 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:29.420 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1489022302 0 0) 0x7f177c004eb0 con 0x7f179014fd10 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f179014adf0 con 0x7f179014fd10 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2773722928 0 0) 0x7f179013d7e0 con 0x7f17900b1f20 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f177c004eb0 con 0x7f17900b1f20 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f17800017d0 con 0x7f179014fd10 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): dump: 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.421 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1788001810 con 0x7f17900b1f20 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.437+0000 7f17777fe640 10 monclient(hunting): dump: 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:29.422 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 357150754 0 0) 0x7f179014adf0 con 0x7f179014fd10 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 >> v1:172.21.15.32:6789/0 conn(0x7f1790146400 legacy=0x7f179013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 >> v1:172.21.15.3:6789/0 conn(0x7f17900b1f20 legacy=0x7f17900b6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 1 monclient: found mon.c 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:29.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f179013d970 con 0x7f179014fd10 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f17777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 10 monclient: _renew_subs 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17901495e0 con 0x7f179014fd10 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 10 monclient: _renew_subs 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17901df040 con 0x7f179014fd10 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.439+0000 7f1795081640 1 librados: init done 2024-03-29T11:16:29.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.441+0000 7f1795081640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1780001ce0 con 0x7f179014fd10 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1780004980 con 0x7f179014fd10 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f17777fe640 10 monclient: dump: 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:29.426 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.442+0000 7f178cd99640 4 set_mon_vals no callback set 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.443+0000 7f1795081640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.443+0000 7f1795081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.443+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17900b74e0 con 0x7f179014fd10 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.444+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f1780004ae0 con 0x7f179014fd10 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f17800849e0 con 0x7f179014fd10 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 10 client.4298.objecter ms_dispatch 0x7f1790000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:16:29.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f178ed9d640 10 monclient: get_auth_request con 0x7f177c06e220 auth_method 0 2024-03-29T11:16:29.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.445+0000 7f17777fe640 10 client.4298.objecter ms_handle_connect 0x7f177c06e220 2024-03-29T11:16:29.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.452+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17800500f0 con 0x7f179014fd10 2024-03-29T11:16:29.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.452+0000 7f17777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:29.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.452+0000 7f17777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:29.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.788+0000 7f1795081640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:29.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.788+0000 7f1795081640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:29.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.788+0000 7f1795081640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:29.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.788+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f179013db00 con 0x7f179014fd10 2024-03-29T11:16:29.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.790+0000 7f17777fe640 1 -- 172.21.15.3:0/1600265137 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f17800521f0 con 0x7f179014fd10 2024-03-29T11:16:29.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.790+0000 7f17777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:29.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.790+0000 7f17777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:29.769 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:29.769 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:29.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.795+0000 7f1795081640 10 librados: watch_flush enter 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 10 librados: watch_flush exit 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 >> v1:172.21.15.3:6816/844689376 conn(0x7f177c06e220 legacy=0x7f177c0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 10 monclient: shutdown 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.797+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 >> v1:172.21.15.3:6790/0 conn(0x7f179014fd10 legacy=0x7f17901de930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.798+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 shutdown_connections 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.798+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 >> 172.21.15.3:0/1600265137 conn(0x7f1790066a00 msgr2=0x7f1790146860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.798+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 shutdown_connections 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.798+0000 7f1795081640 1 -- 172.21.15.3:0/1600265137 wait complete. 2024-03-29T11:16:29.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:29.798+0000 7f1795081640 1 librados: shutdown 2024-03-29T11:16:29.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:36.603 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.775+0000 7fbbaa10e640 10 monclient: get_monmap_and_config 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.775+0000 7fbbaa10e640 10 monclient: build_initial_monmap 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.775+0000 7fbbaa10e640 10 monclient: monmap: 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:36.752 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:36.776119+0000 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:36.776119+0000 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:36.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.776+0000 7fbbaa10e640 1 Processor -- start 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.776+0000 7fbbaa10e640 1 -- start start 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.776+0000 7fbbaa10e640 10 monclient: init 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-b con 0x7fbba40b1cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-c con 0x7fbba40b2090 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-a con 0x7fbba4146fc0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:36.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba406eba0 con 0x7fbba4146fc0 2024-03-29T11:16:36.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba406ee10 con 0x7fbba40b2090 2024-03-29T11:16:36.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba40b4380 con 0x7fbba40b1cb0 2024-03-29T11:16:36.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:36.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.777+0000 7fbbaa10e640 10 monclient(hunting): authenticate will time out at 924.031231s 2024-03-29T11:16:36.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.782+0000 7fbba2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbba40b2090 0x7fbba41468b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:57023/0 (socket says 172.21.15.3:57023) 2024-03-29T11:16:36.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.782+0000 7fbba2ffd640 1 -- 172.21.15.3:0/4241150877 learned_addr learned my addr 172.21.15.3:0/4241150877 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:36.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198195120 0 0) 0x7fbba40b4380 con 0x7fbba40b1cb0 2024-03-29T11:16:36.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:36.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:36.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c003520 con 0x7fbba40b1cb0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94972134 0 0) 0x7fbb8c003520 con 0x7fbba40b1cb0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 >> v1:172.21.15.32:6789/0 conn(0x7fbba40b2090 legacy=0x7fbba41468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 >> v1:172.21.15.3:6789/0 conn(0x7fbba4146fc0 legacy=0x7fbba4151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 monclient: found mon.noname-b 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba41407d0 con 0x7fbba40b1cb0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.785+0000 7fbba27fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:36.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbb94001690 con 0x7fbba40b1cb0 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbbaa10e640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbbaa10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 10 monclient: dump: 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.766 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbbaa10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 1 -- 172.21.15.3:0/4241150877 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb94001920 con 0x7fbba40b1cb0 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbba27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbbaa10e640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.786+0000 7fbbaa10e640 4 set_mon_vals no callback set 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.787+0000 7fbbaa10e640 10 monclient: shutdown 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.787+0000 7fbbaa10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:36.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.787+0000 7fbbaa10e640 1 -- 172.21.15.3:0/4241150877 >> v1:172.21.15.3:6790/0 conn(0x7fbba40b1cb0 legacy=0x7fbba41400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.787+0000 7fbbaa10e640 1 -- 172.21.15.3:0/4241150877 shutdown_connections 2024-03-29T11:16:36.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.787+0000 7fbbaa10e640 1 -- 172.21.15.3:0/4241150877 >> 172.21.15.3:0/4241150877 conn(0x7fbba4066a00 msgr2=0x7fbba4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:36.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.789+0000 7fbbaa10e640 1 -- 172.21.15.3:0/4241150877 shutdown_connections 2024-03-29T11:16:36.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.789+0000 7fbbaa10e640 1 -- 172.21.15.3:0/4241150877 wait complete. 2024-03-29T11:16:36.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.790+0000 7fbbaa10e640 10 monclient: build_initial_monmap 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.794+0000 7fbbaa10e640 10 monclient: monmap: 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:36.772 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:36.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.795+0000 7fbbaa10e640 1 librados: starting msgr at 2024-03-29T11:16:36.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.795+0000 7fbbaa10e640 1 librados: starting objecter 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.796+0000 7fbbaa10e640 1 Processor -- start 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.796+0000 7fbbaa10e640 1 -- start start 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.796+0000 7fbbaa10e640 1 librados: setting wanted keys 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.796+0000 7fbbaa10e640 1 librados: calling monclient init 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.796+0000 7fbbaa10e640 10 monclient: init 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.797+0000 7fbbaa10e640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:36.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.797+0000 7fbbaa10e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:36.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-a con 0x7fbba40b1cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:36.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-b con 0x7fbba4146fc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:36.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 10 monclient(hunting): picked mon.noname-c con 0x7fbba4121200 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:36.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba41222e0 con 0x7fbba40b1cb0 2024-03-29T11:16:36.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba41205d0 con 0x7fbba4146fc0 2024-03-29T11:16:36.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba41487a0 con 0x7fbba4121200 2024-03-29T11:16:36.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:36.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.798+0000 7fbbaa10e640 10 monclient(hunting): authenticate will time out at 924.052652s 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.801+0000 7fbba2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbba4146fc0 0x7fbba413f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:57165/0 (socket says 172.21.15.3:57165) 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.801+0000 7fbba2ffd640 1 -- 172.21.15.3:0/1705707293 learned_addr learned my addr 172.21.15.3:0/1705707293 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.801+0000 7fbb8bfff640 10 client.?.objecter ms_handle_connect 0x7fbba4146fc0 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.801+0000 7fbb8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252173744 0 0) 0x7fbba41205d0 con 0x7fbba4146fc0 2024-03-29T11:16:36.779 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:36.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:36.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:36.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:16:36.780 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.802+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb90004eb0 con 0x7fbba4146fc0 2024-03-29T11:16:36.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.804+0000 7fbb8bfff640 10 client.?.objecter ms_handle_connect 0x7fbba4121200 2024-03-29T11:16:36.781 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.804+0000 7fbb8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 client.?.objecter ms_handle_connect 0x7fbba40b1cb0 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 995300494 0 0) 0x7fbb90004eb0 con 0x7fbba4146fc0 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbba41205d0 con 0x7fbba4146fc0 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152972687 0 0) 0x7fbba41487a0 con 0x7fbba4121200 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:36.786 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.806+0000 7fbb8bfff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb90004eb0 con 0x7fbba4121200 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178373587 0 0) 0x7fbba41222e0 con 0x7fbba40b1cb0 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba41487a0 con 0x7fbba40b1cb0 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbb9c001cb0 con 0x7fbba4146fc0 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): dump: 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:36.787 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 332352125 0 0) 0x7fbba41205d0 con 0x7fbba4146fc0 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 >> v1:172.21.15.3:6790/0 conn(0x7fbba4121200 legacy=0x7fbba41e6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 >> v1:172.21.15.3:6789/0 conn(0x7fbba40b1cb0 legacy=0x7fbba4120af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.788 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 monclient: found mon.b 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba4148930 con 0x7fbba4146fc0 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient: _finish_auth 0 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbb8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 10 monclient: _renew_subs 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:16:36.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba41dc1a0 con 0x7fbba4146fc0 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 10 monclient: _renew_subs 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba41db940 con 0x7fbba4146fc0 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.807+0000 7fbbaa10e640 1 librados: init done 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb9c004b60 con 0x7fbba4146fc0 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbb9c004f30 con 0x7fbba4146fc0 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbb8bfff640 10 monclient: dump: 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:36.793 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.808+0000 7fbba1ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.810+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fbb9c005090 con 0x7fbba4146fc0 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.810+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fbb9c084e60 con 0x7fbba4146fc0 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.810+0000 7fbb8bfff640 10 client.4306.objecter ms_dispatch 0x7fbba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.810+0000 7fbb8bfff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:36.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.810+0000 7fbb8bfff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.811+0000 7fbb8bfff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.811+0000 7fbba37fe640 10 monclient: get_auth_request con 0x7fbb9006def0 auth_method 0 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.812+0000 7fbb8bfff640 10 client.4306.objecter ms_handle_connect 0x7fbb9006def0 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.813+0000 7fbbaa10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.813+0000 7fbbaa10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.813+0000 7fbbaa10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:16:36.795 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.813+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb84005080 con 0x7fbba4146fc0 2024-03-29T11:16:36.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.822+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb9c050740 con 0x7fbba4146fc0 2024-03-29T11:16:36.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.822+0000 7fbb8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:36.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:36.822+0000 7fbb8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:37.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.157+0000 7fbbaa10e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.157+0000 7fbbaa10e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.157+0000 7fbbaa10e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:16:37.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.157+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb84005330 con 0x7fbba4146fc0 2024-03-29T11:16:37.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.159+0000 7fbb8bfff640 1 -- 172.21.15.3:0/1705707293 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fbb9c052670 con 0x7fbba4146fc0 2024-03-29T11:16:37.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.160+0000 7fbb8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:37.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.160+0000 7fbb8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:37.137 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:37.137 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.163+0000 7fbbaa10e640 10 librados: watch_flush enter 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 10 librados: watch_flush exit 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 10 client.4306.objecter successfully canceled tick 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 >> v1:172.21.15.3:6816/844689376 conn(0x7fbb9006def0 legacy=0x7fbb900703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 10 monclient: shutdown 2024-03-29T11:16:37.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 >> v1:172.21.15.32:6789/0 conn(0x7fbba4146fc0 legacy=0x7fbba413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 shutdown_connections 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 >> 172.21.15.3:0/1705707293 conn(0x7fbba4066a00 msgr2=0x7fbba40b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 shutdown_connections 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 -- 172.21.15.3:0/1705707293 wait complete. 2024-03-29T11:16:37.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:37.164+0000 7fbbaa10e640 1 librados: shutdown 2024-03-29T11:16:37.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:43.967 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:44.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.146+0000 7fdb53fff640 10 monclient: get_monmap_and_config 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.146+0000 7fdb53fff640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.146+0000 7fdb53fff640 10 monclient: monmap: 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:44.148641+0000 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:44.148641+0000 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:44.125 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.147+0000 7fdb53fff640 1 Processor -- start 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.147+0000 7fdb53fff640 1 -- start start 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.147+0000 7fdb53fff640 10 monclient: init 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdb540af800 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdb540b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdb54145cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540b0a00 con 0x7fdb54145cb0 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540b17b0 con 0x7fdb540af800 2024-03-29T11:16:44.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb540aed60 con 0x7fdb540b1cd0 2024-03-29T11:16:44.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.148+0000 7fdb53fff640 10 monclient(hunting): authenticate will time out at 931.403771s 2024-03-29T11:16:44.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.152+0000 7fdb537fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fdb54145cb0 0x7fdb5414d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37205/0 (socket says 172.21.15.3:37205) 2024-03-29T11:16:44.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.152+0000 7fdb537fe640 1 -- 172.21.15.3:0/1307050648 learned_addr learned my addr 172.21.15.3:0/1307050648 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:44.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2393050051 0 0) 0x7fdb540b0a00 con 0x7fdb54145cb0 2024-03-29T11:16:44.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb3c003520 con 0x7fdb54145cb0 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622359718 0 0) 0x7fdb540aed60 con 0x7fdb540b1cd0 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:16:44.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.154+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb540b0a00 con 0x7fdb540b1cd0 2024-03-29T11:16:44.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4076324678 0 0) 0x7fdb3c003520 con 0x7fdb54145cb0 2024-03-29T11:16:44.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 >> v1:172.21.15.3:6790/0 conn(0x7fdb540b1cd0 legacy=0x7fdb5413f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 >> v1:172.21.15.32:6789/0 conn(0x7fdb540af800 legacy=0x7fdb541455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 monclient: found mon.noname-a 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb540aeef0 con 0x7fdb54145cb0 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb4c001a10 con 0x7fdb54145cb0 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: dump: 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.138 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb4c001c30 con 0x7fdb54145cb0 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 1 -- 172.21.15.3:0/1307050648 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb4c001f40 con 0x7fdb54145cb0 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.158+0000 7fdb51ffb640 10 monclient: dump: 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:44.139 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:44.140 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.159+0000 7fdb53fff640 5 monclient: authenticate success, global_id 4299 2024-03-29T11:16:44.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.159+0000 7fdb53fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:44.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.159+0000 7fdb53fff640 4 set_mon_vals no callback set 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.160+0000 7fdb53fff640 10 monclient: shutdown 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.160+0000 7fdb53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.160+0000 7fdb53fff640 1 -- 172.21.15.3:0/1307050648 >> v1:172.21.15.3:6789/0 conn(0x7fdb54145cb0 legacy=0x7fdb5414d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.161+0000 7fdb53fff640 1 -- 172.21.15.3:0/1307050648 shutdown_connections 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.161+0000 7fdb53fff640 1 -- 172.21.15.3:0/1307050648 >> 172.21.15.3:0/1307050648 conn(0x7fdb54066a00 msgr2=0x7fdb54066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.161+0000 7fdb53fff640 1 -- 172.21.15.3:0/1307050648 shutdown_connections 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.161+0000 7fdb53fff640 1 -- 172.21.15.3:0/1307050648 wait complete. 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.162+0000 7fdb53fff640 10 monclient: build_initial_monmap 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.162+0000 7fdb53fff640 10 monclient: monmap: 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:44.142 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.162+0000 7fdb53fff640 1 librados: starting msgr at 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.162+0000 7fdb53fff640 1 librados: starting objecter 2024-03-29T11:16:44.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.163+0000 7fdb53fff640 1 Processor -- start 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.163+0000 7fdb53fff640 1 -- start start 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.163+0000 7fdb53fff640 1 librados: setting wanted keys 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.163+0000 7fdb53fff640 1 librados: calling monclient init 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.163+0000 7fdb53fff640 10 monclient: init 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-b con 0x7fdb540af800 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-c con 0x7fdb540b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient(hunting): picked mon.noname-a con 0x7fdb54140c50 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb541ccee0 con 0x7fdb54140c50 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb541cd1c0 con 0x7fdb540af800 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb541cbdc0 con 0x7fdb540b1cd0 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:44.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.164+0000 7fdb53fff640 10 monclient(hunting): authenticate will time out at 931.419207s 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb527fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdb540b1cd0 0x7fdb54140540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47669/0 (socket says 172.21.15.3:47669) 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb527fc640 1 -- 172.21.15.3:0/444140555 learned_addr learned my addr 172.21.15.3:0/444140555 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb540b1cd0 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461885405 0 0) 0x7fdb541cbdc0 con 0x7fdb540b1cd0 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:16:44.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.168+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb40004eb0 con 0x7fdb540b1cd0 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3063734989 0 0) 0x7fdb40004eb0 con 0x7fdb540b1cd0 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb541cbdc0 con 0x7fdb540b1cd0 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb48001940 con 0x7fdb540b1cd0 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:44.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.171+0000 7fdb3b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.172+0000 7fdb3b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb54140c50 2024-03-29T11:16:44.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.172+0000 7fdb3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213906712 0 0) 0x7fdb541ccee0 con 0x7fdb54140c50 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb40004eb0 con 0x7fdb54140c50 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700544568 0 0) 0x7fdb40004eb0 con 0x7fdb54140c50 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb541ccee0 con 0x7fdb54140c50 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb4c0018d0 con 0x7fdb54140c50 2024-03-29T11:16:44.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.173+0000 7fdb3b7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2281515843 0 0) 0x7fdb541ccee0 con 0x7fdb54140c50 2024-03-29T11:16:44.154 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 >> v1:172.21.15.3:6790/0 conn(0x7fdb540b1cd0 legacy=0x7fdb54140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 >> v1:172.21.15.32:6789/0 conn(0x7fdb540af800 legacy=0x7fdb541cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 monclient: found mon.a 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54142ef0 con 0x7fdb54140c50 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 client.?.objecter ms_handle_connect 0x7fdb540af800 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb4c001a30 con 0x7fdb54140c50 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb4c002100 con 0x7fdb54140c50 2024-03-29T11:16:44.155 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.174+0000 7fdb517fa640 4 set_mon_vals no callback set 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 5 monclient: authenticate success, global_id 4302 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 10 monclient: _renew_subs 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb541cc2e0 con 0x7fdb54140c50 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 10 monclient: _renew_subs 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb541e7d00 con 0x7fdb54140c50 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.175+0000 7fdb53fff640 1 librados: init done 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.176+0000 7fdb3b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.176+0000 7fdb3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:44.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.176+0000 7fdb3b7fe640 10 monclient: dump: 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:44.158 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.178+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fdb4c01ae20 con 0x7fdb54140c50 2024-03-29T11:16:44.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.178+0000 7fdb53fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.178+0000 7fdb53fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.178+0000 7fdb53fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.178+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb541cbf50 con 0x7fdb54140c50 2024-03-29T11:16:44.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.179+0000 7fdb52ffd640 10 monclient: get_auth_request con 0x7fdb4006deb0 auth_method 0 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.179+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fdb4c084c50 con 0x7fdb54140c50 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.180+0000 7fdb3b7fe640 10 client.4302.objecter ms_dispatch 0x7fdb54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.180+0000 7fdb3b7fe640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.180+0000 7fdb3b7fe640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.180+0000 7fdb3b7fe640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-29T11:16:44.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.180+0000 7fdb3b7fe640 10 client.4302.objecter ms_handle_connect 0x7fdb4006deb0 2024-03-29T11:16:44.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.187+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb4c050530 con 0x7fdb54140c50 2024-03-29T11:16:44.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.187+0000 7fdb3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:44.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.187+0000 7fdb3b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:44.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.515+0000 7fdb53fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:44.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.515+0000 7fdb53fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:44.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.515+0000 7fdb53fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:16:44.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.515+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb5406a400 con 0x7fdb54140c50 2024-03-29T11:16:44.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.520+0000 7fdb3b7fe640 1 -- 172.21.15.3:0/444140555 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fdb4c052460 con 0x7fdb54140c50 2024-03-29T11:16:44.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.520+0000 7fdb3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:44.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.520+0000 7fdb3b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:44.500 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:44.500 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:44.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.525+0000 7fdb53fff640 10 librados: watch_flush enter 2024-03-29T11:16:44.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 10 librados: watch_flush exit 2024-03-29T11:16:44.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-29T11:16:44.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 10 client.4302.objecter successfully canceled tick 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 >> v1:172.21.15.3:6816/844689376 conn(0x7fdb4006deb0 legacy=0x7fdb40070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 10 monclient: shutdown 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.526+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 >> v1:172.21.15.3:6789/0 conn(0x7fdb54140c50 legacy=0x7fdb541e65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.527+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 shutdown_connections 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.527+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 >> 172.21.15.3:0/444140555 conn(0x7fdb54066a00 msgr2=0x7fdb540b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:44.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.527+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 shutdown_connections 2024-03-29T11:16:44.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.528+0000 7fdb53fff640 1 -- 172.21.15.3:0/444140555 wait complete. 2024-03-29T11:16:44.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:44.529+0000 7fdb53fff640 1 librados: shutdown 2024-03-29T11:16:44.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:51.335 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:51.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.513+0000 7f34da031640 10 monclient: get_monmap_and_config 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.513+0000 7f34da031640 10 monclient: build_initial_monmap 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.513+0000 7f34da031640 10 monclient: monmap: 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:51.515685+0000 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:51.515685+0000 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:51.492 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:51.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.515+0000 7f34da031640 1 Processor -- start 2024-03-29T11:16:51.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.515+0000 7f34da031640 1 -- start start 2024-03-29T11:16:51.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient: init 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient(hunting): picked mon.noname-a con 0x7f34d40b3cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient(hunting): picked mon.noname-c con 0x7f34d40aecd0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient(hunting): picked mon.noname-b con 0x7f34d40b1070 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40afb60 con 0x7f34d40b3cb0 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40b4cf0 con 0x7f34d40aecd0 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d413fae0 con 0x7f34d40b1070 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:51.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.516+0000 7f34da031640 10 monclient(hunting): authenticate will time out at 938.771954s 2024-03-29T11:16:51.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d3fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f34d40b1070 0x7f34d414d890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58705/0 (socket says 172.21.15.3:58705) 2024-03-29T11:16:51.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d3fff640 1 -- 172.21.15.3:0/4149503360 learned_addr learned my addr 172.21.15.3:0/4149503360 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:51.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200284963 0 0) 0x7f34d413fae0 con 0x7f34d40b1070 2024-03-29T11:16:51.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:51.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:51.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:51.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:16:51.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.521+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c0003520 con 0x7f34d40b1070 2024-03-29T11:16:51.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140157128 0 0) 0x7f34d40b4cf0 con 0x7f34d40aecd0 2024-03-29T11:16:51.500 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:51.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:51.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:51.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:16:51.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d413fae0 con 0x7f34d40aecd0 2024-03-29T11:16:51.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 265114089 0 0) 0x7f34c0003520 con 0x7f34d40b1070 2024-03-29T11:16:51.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 >> v1:172.21.15.32:6789/0 conn(0x7f34d40aecd0 legacy=0x7f34d413f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.522+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 >> v1:172.21.15.3:6789/0 conn(0x7f34d40b3cb0 legacy=0x7f34d40b45e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:51.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:51.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 1 monclient: found mon.noname-b 2024-03-29T11:16:51.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d413fc70 con 0x7f34d40b1070 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f34cc001610 con 0x7f34d40b1070 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.523+0000 7f34d27fc640 10 monclient: dump: 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.504 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.524+0000 7f34da031640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:16:51.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.524+0000 7f34da031640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:51.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.525+0000 7f34d27fc640 1 -- 172.21.15.3:0/4149503360 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34cc0017f0 con 0x7f34d40b1070 2024-03-29T11:16:51.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.525+0000 7f34d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:51.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.525+0000 7f34da031640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:51.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.525+0000 7f34da031640 4 set_mon_vals no callback set 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.526+0000 7f34da031640 10 monclient: shutdown 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.526+0000 7f34da031640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.526+0000 7f34da031640 1 -- 172.21.15.3:0/4149503360 >> v1:172.21.15.3:6790/0 conn(0x7f34d40b1070 legacy=0x7f34d414d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.526+0000 7f34da031640 1 -- 172.21.15.3:0/4149503360 shutdown_connections 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.526+0000 7f34da031640 1 -- 172.21.15.3:0/4149503360 >> 172.21.15.3:0/4149503360 conn(0x7f34d4066a00 msgr2=0x7f34d4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:51.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.528+0000 7f34da031640 1 -- 172.21.15.3:0/4149503360 shutdown_connections 2024-03-29T11:16:51.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.530+0000 7f34da031640 1 -- 172.21.15.3:0/4149503360 wait complete. 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.531+0000 7f34da031640 10 monclient: build_initial_monmap 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.531+0000 7f34da031640 10 monclient: monmap: 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.531+0000 7f34da031640 1 librados: starting msgr at 2024-03-29T11:16:51.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.531+0000 7f34da031640 1 librados: starting objecter 2024-03-29T11:16:51.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.531+0000 7f34da031640 1 Processor -- start 2024-03-29T11:16:51.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.532+0000 7f34da031640 1 -- start start 2024-03-29T11:16:51.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.532+0000 7f34da031640 1 librados: setting wanted keys 2024-03-29T11:16:51.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.532+0000 7f34da031640 1 librados: calling monclient init 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.532+0000 7f34da031640 10 monclient: init 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient(hunting): picked mon.noname-c con 0x7f34d40aecd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient(hunting): picked mon.noname-b con 0x7f34d40b3cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient(hunting): picked mon.noname-a con 0x7f34d4071e50 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d41ca0f0 con 0x7f34d4071e50 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d41c9110 con 0x7f34d40b3cb0 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d41c92a0 con 0x7f34d40aecd0 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:51.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.533+0000 7f34da031640 10 monclient(hunting): authenticate will time out at 938.788156s 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.536+0000 7f34d37fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f34d40aecd0 0x7f34d41c8a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58837/0 (socket says 172.21.15.3:58837) 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.536+0000 7f34d37fe640 1 -- 172.21.15.3:0/2036014475 learned_addr learned my addr 172.21.15.3:0/2036014475 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter ms_handle_connect 0x7f34d40aecd0 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833571817 0 0) 0x7f34d41c92a0 con 0x7f34d40aecd0 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b4004eb0 con 0x7f34d40aecd0 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter ms_handle_connect 0x7f34d4071e50 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:51.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter ms_handle_connect 0x7f34d40b3cb0 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.539+0000 7f34cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049308865 0 0) 0x7f34b4004eb0 con 0x7f34d40aecd0 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d41c92a0 con 0x7f34d40aecd0 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f34bc0017d0 con 0x7f34d40aecd0 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): dump: 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:51.522 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272429775 0 0) 0x7f34d41c9110 con 0x7f34d40b3cb0 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b4004eb0 con 0x7f34d40b3cb0 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294671677 0 0) 0x7f34d41ca0f0 con 0x7f34d4071e50 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:51.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d41c9110 con 0x7f34d4071e50 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2318902310 0 0) 0x7f34d41c92a0 con 0x7f34d40aecd0 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 >> v1:172.21.15.32:6789/0 conn(0x7f34d40b3cb0 legacy=0x7f34d41e3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 >> v1:172.21.15.3:6789/0 conn(0x7f34d4071e50 legacy=0x7f34d41e4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 monclient: found mon.c 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d41f25f0 con 0x7f34d40aecd0 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient: _finish_auth 0 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:16:51.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 10 monclient: _renew_subs 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d41ef5c0 con 0x7f34d40aecd0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 10 monclient: _renew_subs 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d41efaf0 con 0x7f34d40aecd0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.540+0000 7f34da031640 1 librados: init done 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34bc001d20 con 0x7f34d40aecd0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f34bc004a00 con 0x7f34d40aecd0 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.541+0000 7f34cbfff640 10 monclient: dump: 2024-03-29T11:16:51.525 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:51.526 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.544+0000 7f34da031640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.547+0000 7f34da031640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.547+0000 7f34da031640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.547+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b8005080 con 0x7f34d40aecd0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.547+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f34bc004b60 con 0x7f34d40aecd0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f34bc084a30 con 0x7f34d40aecd0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34cbfff640 10 client.4316.objecter ms_dispatch 0x7f34d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34cbfff640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34cbfff640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34cbfff640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.548+0000 7f34d2ffd640 10 monclient: get_auth_request con 0x7f34b406dea0 auth_method 0 2024-03-29T11:16:51.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.549+0000 7f34cbfff640 10 client.4316.objecter ms_handle_connect 0x7f34b406dea0 2024-03-29T11:16:51.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.557+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34bc050140 con 0x7f34d40aecd0 2024-03-29T11:16:51.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.557+0000 7f34cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:51.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.557+0000 7f34cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:51.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.886+0000 7f34da031640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:51.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.886+0000 7f34da031640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:51.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.886+0000 7f34da031640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:51.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.886+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34b8005330 con 0x7f34d40aecd0 2024-03-29T11:16:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.889+0000 7f34cbfff640 1 -- 172.21.15.3:0/2036014475 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f34bc052240 con 0x7f34d40aecd0 2024-03-29T11:16:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.889+0000 7f34cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:51.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.889+0000 7f34cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:51.867 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:51.868 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:51.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 10 librados: watch_flush enter 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 10 librados: watch_flush exit 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 >> v1:172.21.15.3:6816/844689376 conn(0x7f34b406dea0 legacy=0x7f34b4070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 10 monclient: shutdown 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 >> v1:172.21.15.3:6790/0 conn(0x7f34d40aecd0 legacy=0x7f34d41c8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 shutdown_connections 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 >> 172.21.15.3:0/2036014475 conn(0x7f34d4066a00 msgr2=0x7f34d413f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 shutdown_connections 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.893+0000 7f34da031640 1 -- 172.21.15.3:0/2036014475 wait complete. 2024-03-29T11:16:51.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:51.894+0000 7f34da031640 1 librados: shutdown 2024-03-29T11:16:51.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:58.696 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.863+0000 7f42e9286640 10 monclient: get_monmap_and_config 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.863+0000 7f42e9286640 10 monclient: build_initial_monmap 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.863+0000 7f42e9286640 10 monclient: monmap: 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:16:58.863940+0000 2024-03-29T11:16:58.840 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:16:58.863940+0000 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:16:58.841 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.863+0000 7f42e9286640 1 Processor -- start 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.864+0000 7f42e9286640 1 -- start start 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.864+0000 7f42e9286640 10 monclient: init 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.864+0000 7f42e9286640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:16:58.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-b con 0x7f42e40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-c con 0x7f42e40b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-a con 0x7f42e40b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e40aab60 con 0x7f42e40b3560 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e2ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f42e40b28d0 0x7f42e413bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54294/0 (socket says 172.21.15.3:54294) 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e2ffd640 1 -- 172.21.15.3:0/1074059508 learned_addr learned my addr 172.21.15.3:0/1074059508 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e40aae40 con 0x7f42e40b2cb0 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e40ab120 con 0x7f42e40b28d0 2024-03-29T11:16:58.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e9286640 10 monclient(hunting): authenticate will time out at 946.119506s 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575402943 0 0) 0x7f42e40ab120 con 0x7f42e40b28d0 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.865+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42cc003520 con 0x7f42e40b28d0 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756137903 0 0) 0x7f42e40aab60 con 0x7f42e40b3560 2024-03-29T11:16:58.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e40ab120 con 0x7f42e40b3560 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3074799098 0 0) 0x7f42e40ab120 con 0x7f42e40b3560 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 >> v1:172.21.15.3:6790/0 conn(0x7f42e40b28d0 legacy=0x7f42e413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 >> v1:172.21.15.32:6789/0 conn(0x7f42e40b2cb0 legacy=0x7f42e4142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:16:58.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e413c380 con 0x7f42e40b3560 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42dc0019f0 con 0x7f42e40b3560 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:58.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 10 monclient: dump: 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.850 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42dc001b50 con 0x7f42e40b3560 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e9286640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.866+0000 7f42e9286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e1ffb640 1 -- 172.21.15.3:0/1074059508 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42dc001fa0 con 0x7f42e40b3560 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e1ffb640 10 monclient: dump: 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.851 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e9286640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.867+0000 7f42e9286640 4 set_mon_vals no callback set 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 10 monclient: shutdown 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 >> v1:172.21.15.3:6789/0 conn(0x7f42e40b3560 legacy=0x7f42e41428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 shutdown_connections 2024-03-29T11:16:58.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 >> 172.21.15.3:0/1074059508 conn(0x7f42e4066a00 msgr2=0x7f42e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 shutdown_connections 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.868+0000 7f42e9286640 1 -- 172.21.15.3:0/1074059508 wait complete. 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.869+0000 7f42e9286640 10 monclient: build_initial_monmap 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.869+0000 7f42e9286640 10 monclient: monmap: 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:58.857 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 librados: starting msgr at 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 librados: starting objecter 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 Processor -- start 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 -- start start 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 librados: setting wanted keys 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 1 librados: calling monclient init 2024-03-29T11:16:58.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.870+0000 7f42e9286640 10 monclient: init 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.871+0000 7f42e9286640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.871+0000 7f42e9286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.871+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-b con 0x7f42e40b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.871+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-a con 0x7f42e4070f90 addr v1:172.21.15.3:6789/0 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.871+0000 7f42e9286640 10 monclient(hunting): picked mon.noname-c con 0x7f42e412e170 addr v1:172.21.15.3:6790/0 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e27fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f42e4070f90 0x7f42e4144d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:57148/0 (socket says 172.21.15.3:57148) 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e9286640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e4144190 con 0x7f42e4070f90 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e9286640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e41437b0 con 0x7f42e40b28d0 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e9286640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e4143a90 con 0x7f42e412e170 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f42e40b28d0 0x7f42e4144670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:55654/0 (socket says 172.21.15.3:55654) 2024-03-29T11:16:58.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e9286640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e2ffd640 1 -- 172.21.15.3:0/3080978358 learned_addr learned my addr 172.21.15.3:0/3080978358 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42e9286640 10 monclient(hunting): authenticate will time out at 946.126159s 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter ms_handle_connect 0x7f42e40b28d0 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter ms_handle_connect 0x7f42e4070f90 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter ms_handle_connect 0x7f42e412e170 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717262608 0 0) 0x7f42e41437b0 con 0x7f42e40b28d0 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:58.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d0004eb0 con 0x7f42e40b28d0 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910457439 0 0) 0x7f42e4144190 con 0x7f42e4070f90 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e41437b0 con 0x7f42e4070f90 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062107361 0 0) 0x7f42e4143a90 con 0x7f42e412e170 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:58.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.872+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42e4144190 con 0x7f42e412e170 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1518549742 0 0) 0x7f42e41437b0 con 0x7f42e4070f90 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e4143a90 con 0x7f42e4070f90 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42d8003df0 con 0x7f42e4070f90 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.862 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1158127028 0 0) 0x7f42d0004eb0 con 0x7f42e40b28d0 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42e41437b0 con 0x7f42e40b28d0 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2997472557 0 0) 0x7f42e4144190 con 0x7f42e412e170 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42d0006f70 con 0x7f42e412e170 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42dc001e60 con 0x7f42e412e170 2024-03-29T11:16:58.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.864 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42d4001a70 con 0x7f42e40b28d0 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.873+0000 7f42cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.865 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 8985034 0 0) 0x7f42d0006f70 con 0x7f42e412e170 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 >> v1:172.21.15.32:6789/0 conn(0x7f42e40b28d0 legacy=0x7f42e4144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 >> v1:172.21.15.3:6789/0 conn(0x7f42e4070f90 legacy=0x7f42e4144d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 monclient: found mon.c 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4143d70 con 0x7f42e412e170 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:58.866 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 10 monclient: _renew_subs 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e41e77d0 con 0x7f42e412e170 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 10 monclient: _renew_subs 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e41e7b40 con 0x7f42e412e170 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e9286640 1 librados: init done 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42dc001fc0 con 0x7f42e412e170 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f42dc0012e0 con 0x7f42e412e170 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:58.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42cb7fe640 10 monclient: dump: 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.874+0000 7f42e17fa640 4 set_mon_vals no callback set 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.875+0000 7f42e9286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:58.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.875+0000 7f42e9286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.875+0000 7f42e9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.875+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c4005080 con 0x7f42e412e170 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f42dc019c30 con 0x7f42e412e170 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f42dc0537a0 con 0x7f42e412e170 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 10 client.4322.objecter ms_dispatch 0x7f42e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.877+0000 7f42cb7fe640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.878+0000 7f42e2ffd640 10 monclient: get_auth_request con 0x7f42d006e3a0 auth_method 0 2024-03-29T11:16:58.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.878+0000 7f42cb7fe640 10 client.4322.objecter ms_handle_connect 0x7f42d006e3a0 2024-03-29T11:16:58.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.887+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42dc04fa40 con 0x7f42e412e170 2024-03-29T11:16:58.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.887+0000 7f42cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:58.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:58.887+0000 7f42cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:59.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.292+0000 7f42e9286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.293+0000 7f42e9286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.293+0000 7f42e9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:16:59.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.293+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42c4005330 con 0x7f42e412e170 2024-03-29T11:16:59.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.294+0000 7f42cb7fe640 1 -- 172.21.15.3:0/3080978358 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f42dc04fa40 con 0x7f42e412e170 2024-03-29T11:16:59.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.294+0000 7f42cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:59.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.294+0000 7f42cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:59.271 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:16:59.272 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 10 librados: watch_flush enter 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 10 librados: watch_flush exit 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 10 client.4322.objecter successfully canceled tick 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 >> v1:172.21.15.3:6816/844689376 conn(0x7f42d006e3a0 legacy=0x7f42d0070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 10 monclient: shutdown 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 >> v1:172.21.15.3:6790/0 conn(0x7f42e412e170 legacy=0x7f42e41e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 shutdown_connections 2024-03-29T11:16:59.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.298+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 >> 172.21.15.3:0/3080978358 conn(0x7f42e4066a00 msgr2=0x7f42e41417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:59.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.299+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 shutdown_connections 2024-03-29T11:16:59.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.299+0000 7f42e9286640 1 -- 172.21.15.3:0/3080978358 wait complete. 2024-03-29T11:16:59.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:16:59.299+0000 7f42e9286640 1 librados: shutdown 2024-03-29T11:16:59.293 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:06.099 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.261+0000 7f903c73d640 10 monclient: get_monmap_and_config 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.261+0000 7f903c73d640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.261+0000 7f903c73d640 10 monclient: monmap: 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:06.262336+0000 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:06.262336+0000 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:06.239 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 1 Processor -- start 2024-03-29T11:17:06.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 1 -- start start 2024-03-29T11:17:06.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient: init 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-b con 0x7f90340b4b40 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-c con 0x7f90340b4f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-a con 0x7f90340a9a90 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340aabd0 con 0x7f90340a9a90 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340aaeb0 con 0x7f90340b4f20 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90340ab190 con 0x7f90340b4b40 2024-03-29T11:17:06.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.262+0000 7f903c73d640 10 monclient(hunting): authenticate will time out at 953.516981s 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f903a4d9640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f90340b4b40 0x7f903413be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:45978/0 (socket says 172.21.15.3:45978) 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f903a4d9640 1 -- 172.21.15.3:0/848105467 learned_addr learned my addr 172.21.15.3:0/848105467 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393461604 0 0) 0x7f90340ab190 con 0x7f90340b4b40 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.263+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9020003520 con 0x7f90340b4b40 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705531184 0 0) 0x7f90340aaeb0 con 0x7f90340b4f20 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:17:06.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.264+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9020003840 con 0x7f90340b4f20 2024-03-29T11:17:06.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.265+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2487985939 0 0) 0x7f9020003520 con 0x7f90340b4b40 2024-03-29T11:17:06.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.265+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 >> v1:172.21.15.32:6789/0 conn(0x7f90340b4f20 legacy=0x7f90341455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.265+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 >> v1:172.21.15.3:6789/0 conn(0x7f90340a9a90 legacy=0x7f9034145d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 1 monclient: found mon.noname-b 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90340b45a0 con 0x7f90340b4b40 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9030001670 con 0x7f90340b4b40 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f90394d7640 10 monclient: dump: 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.245 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f903c73d640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.266+0000 7f903c73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9030001850 con 0x7f90340b4b40 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 1 -- 172.21.15.3:0/848105467 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9030001c60 con 0x7f90340b4b40 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f903c73d640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f903c73d640 4 set_mon_vals no callback set 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f90394d7640 10 monclient: dump: 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f903c73d640 10 monclient: shutdown 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f903c73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:06.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.267+0000 7f903c73d640 1 -- 172.21.15.3:0/848105467 >> v1:172.21.15.3:6790/0 conn(0x7f90340b4b40 legacy=0x7f903413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.268+0000 7f903c73d640 1 -- 172.21.15.3:0/848105467 shutdown_connections 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.268+0000 7f903c73d640 1 -- 172.21.15.3:0/848105467 >> 172.21.15.3:0/848105467 conn(0x7f9034066a00 msgr2=0x7f9034066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.268+0000 7f903c73d640 1 -- 172.21.15.3:0/848105467 shutdown_connections 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.268+0000 7f903c73d640 1 -- 172.21.15.3:0/848105467 wait complete. 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 10 monclient: build_initial_monmap 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 10 monclient: monmap: 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:06.249 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 librados: starting msgr at 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 librados: starting objecter 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 Processor -- start 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 -- start start 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 librados: setting wanted keys 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 1 librados: calling monclient init 2024-03-29T11:17:06.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.269+0000 7f903c73d640 10 monclient: init 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.270+0000 7f903c73d640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.270+0000 7f903c73d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-c con 0x7f90340a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-a con 0x7f90340b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 10 monclient(hunting): picked mon.noname-b con 0x7f9034144480 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90341d7180 con 0x7f90340b4b40 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90341d4de0 con 0x7f9034144480 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90341d8050 con 0x7f90340a9a90 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903c73d640 10 monclient(hunting): authenticate will time out at 953.525234s 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903a4d9640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f90340a9a90 0x7f90341d7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:49332/0 (socket says 172.21.15.3:49332) 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.271+0000 7f903a4d9640 1 -- 172.21.15.3:0/4064848351 learned_addr learned my addr 172.21.15.3:0/4064848351 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 client.?.objecter ms_handle_connect 0x7f90340a9a90 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172678849 0 0) 0x7f90341d8050 con 0x7f90340a9a90 2024-03-29T11:17:06.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:06.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:06.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:06.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:17:06.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9018004eb0 con 0x7f90340a9a90 2024-03-29T11:17:06.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3490455423 0 0) 0x7f9018004eb0 con 0x7f90340a9a90 2024-03-29T11:17:06.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.272+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90341d8050 con 0x7f90340a9a90 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9030001c60 con 0x7f90340a9a90 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 10 monclient(hunting): dump: 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1931833701 0 0) 0x7f90341d8050 con 0x7f90340a9a90 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 >> v1:172.21.15.32:6789/0 conn(0x7f9034144480 legacy=0x7f90341e19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 >> v1:172.21.15.3:6789/0 conn(0x7f90340b4b40 legacy=0x7f9034145190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.273+0000 7f902affd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 1 monclient: found mon.c 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90341d7420 con 0x7f90340a9a90 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 10 monclient: _finish_auth 0 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 10 monclient: _renew_subs 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90341458a0 con 0x7f90340a9a90 2024-03-29T11:17:06.256 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 10 monclient: _renew_subs 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90341e32b0 con 0x7f90340a9a90 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f903c73d640 1 librados: init done 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9030001210 con 0x7f90340a9a90 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f9038cd6640 4 set_mon_vals no callback set 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9030004d30 con 0x7f90340a9a90 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.274+0000 7f902affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.275+0000 7f902affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.275+0000 7f902affd640 10 monclient: dump: 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:06.257 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.277+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f9030004e90 con 0x7f90340a9a90 2024-03-29T11:17:06.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.278+0000 7f903c73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.278+0000 7f903c73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.278+0000 7f903c73d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.278+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f901c005080 con 0x7f90340a9a90 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f9039cd8640 10 monclient: get_auth_request con 0x7f901806dc70 auth_method 0 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f9030084c80 con 0x7f90340a9a90 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 10 client.4328.objecter ms_dispatch 0x7f9034000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:06.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:06.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:17:06.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.279+0000 7f902affd640 10 client.4328.objecter ms_handle_connect 0x7f901806dc70 2024-03-29T11:17:06.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.285+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9030050560 con 0x7f90340a9a90 2024-03-29T11:17:06.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.285+0000 7f902affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:06.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.285+0000 7f902affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:06.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.592+0000 7f903c73d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:06.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.592+0000 7f903c73d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:06.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.592+0000 7f903c73d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:06.570 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.593+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f901c005330 con 0x7f90340a9a90 2024-03-29T11:17:06.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.593+0000 7f902affd640 1 -- 172.21.15.3:0/4064848351 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f9030052490 con 0x7f90340a9a90 2024-03-29T11:17:06.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.593+0000 7f902affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:06.571 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:06.571 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:06.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.593+0000 7f902affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.596+0000 7f903c73d640 10 librados: watch_flush enter 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.596+0000 7f903c73d640 10 librados: watch_flush exit 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 >> v1:172.21.15.3:6816/844689376 conn(0x7f901806dc70 legacy=0x7f9018070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 10 monclient: shutdown 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 >> v1:172.21.15.3:6790/0 conn(0x7f90340a9a90 legacy=0x7f90341d7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 shutdown_connections 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 >> 172.21.15.3:0/4064848351 conn(0x7f9034066a00 msgr2=0x7f90340b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 shutdown_connections 2024-03-29T11:17:06.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 -- 172.21.15.3:0/4064848351 wait complete. 2024-03-29T11:17:06.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:06.597+0000 7f903c73d640 1 librados: shutdown 2024-03-29T11:17:06.588 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:13.395 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:13.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.560+0000 7f8932dfe640 10 monclient: get_monmap_and_config 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.560+0000 7f8932dfe640 10 monclient: build_initial_monmap 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.560+0000 7f8932dfe640 10 monclient: monmap: 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:13.561334+0000 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:13.561334+0000 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:13.539 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.562+0000 7f8932dfe640 1 Processor -- start 2024-03-29T11:17:13.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.562+0000 7f8932dfe640 1 -- start start 2024-03-29T11:17:13.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.562+0000 7f8932dfe640 10 monclient: init 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-c con 0x7f892c149c10 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-b con 0x7f892c0a9630 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-a con 0x7f892c14a450 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0aad40 con 0x7f892c14a450 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c14a8e0 con 0x7f892c149c10 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c0abb60 con 0x7f892c0a9630 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:13.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.563+0000 7f8932dfe640 10 monclient(hunting): authenticate will time out at 960.817802s 2024-03-29T11:17:13.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.566+0000 7f8930b9a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f892c149c10 0x7f892c0ae7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:37876/0 (socket says 172.21.15.3:37876) 2024-03-29T11:17:13.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.566+0000 7f8930b9a640 1 -- 172.21.15.3:0/4128484644 learned_addr learned my addr 172.21.15.3:0/4128484644 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:13.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113977866 0 0) 0x7f892c14a8e0 con 0x7f892c149c10 2024-03-29T11:17:13.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:13.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8914003520 con 0x7f892c149c10 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095650741 0 0) 0x7f892c0aad40 con 0x7f892c14a450 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c14a8e0 con 0x7f892c14a450 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794701459 0 0) 0x7f892c0abb60 con 0x7f892c0a9630 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:13.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f892c0aad40 con 0x7f892c0a9630 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.568+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3957110086 0 0) 0x7f8914003520 con 0x7f892c149c10 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 >> v1:172.21.15.3:6790/0 conn(0x7f892c0a9630 legacy=0x7f892c143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 >> v1:172.21.15.3:6789/0 conn(0x7f892c14a450 legacy=0x7f892c151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c0aa8d0 con 0x7f892c149c10 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8920001690 con 0x7f892c149c10 2024-03-29T11:17:13.548 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: dump: 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f8932dfe640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f8932dfe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:13.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89200017f0 con 0x7f892c149c10 2024-03-29T11:17:13.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f892b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f8932dfe640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.569+0000 7f8932dfe640 4 set_mon_vals no callback set 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f892b7fe640 1 -- 172.21.15.3:0/4128484644 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8920001c00 con 0x7f892c149c10 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f892b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f892b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:13.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f892b7fe640 10 monclient: dump: 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f8932dfe640 10 monclient: shutdown 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f8932dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.570+0000 7f8932dfe640 1 -- 172.21.15.3:0/4128484644 >> v1:172.21.15.32:6789/0 conn(0x7f892c149c10 legacy=0x7f892c0ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.571+0000 7f8932dfe640 1 -- 172.21.15.3:0/4128484644 shutdown_connections 2024-03-29T11:17:13.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.571+0000 7f8932dfe640 1 -- 172.21.15.3:0/4128484644 >> 172.21.15.3:0/4128484644 conn(0x7f892c0b3c10 msgr2=0x7f892c0b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.571+0000 7f8932dfe640 1 -- 172.21.15.3:0/4128484644 shutdown_connections 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.571+0000 7f8932dfe640 1 -- 172.21.15.3:0/4128484644 wait complete. 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.572+0000 7f8932dfe640 10 monclient: build_initial_monmap 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.572+0000 7f8932dfe640 10 monclient: monmap: 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:13.554 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.572+0000 7f8932dfe640 1 librados: starting msgr at 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.572+0000 7f8932dfe640 1 librados: starting objecter 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.573+0000 7f8932dfe640 1 Processor -- start 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.573+0000 7f8932dfe640 1 -- start start 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.573+0000 7f8932dfe640 1 librados: setting wanted keys 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.573+0000 7f8932dfe640 1 librados: calling monclient init 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.573+0000 7f8932dfe640 10 monclient: init 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:13.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-c con 0x7f892c0a9630 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-a con 0x7f892c149c10 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient(hunting): picked mon.noname-b con 0x7f892c14a450 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c1443c0 con 0x7f892c149c10 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c1446a0 con 0x7f892c14a450 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892c144980 con 0x7f892c0a9630 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.574+0000 7f8932dfe640 10 monclient(hunting): authenticate will time out at 960.828186s 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f892bfff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f892c149c10 0x7f892c139760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:56941/0 (socket says 172.21.15.3:56941) 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f892bfff640 1 -- 172.21.15.3:0/1255087731 learned_addr learned my addr 172.21.15.3:0/1255087731 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 10 client.?.objecter ms_handle_connect 0x7f892c149c10 2024-03-29T11:17:13.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960896627 0 0) 0x7f892c1443c0 con 0x7f892c149c10 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.577+0000 7f89297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918004eb0 con 0x7f892c149c10 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530693177 0 0) 0x7f8918004eb0 con 0x7f892c149c10 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f892c1443c0 con 0x7f892c149c10 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f891c001a10 con 0x7f892c149c10 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 monclient(hunting): dump: 2024-03-29T11:17:13.560 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.578+0000 7f89297fa640 10 client.?.objecter ms_handle_connect 0x7f892c14a450 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.579+0000 7f89297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.579+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3705768938 0 0) 0x7f892c1443c0 con 0x7f892c149c10 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.579+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 >> v1:172.21.15.3:6790/0 conn(0x7f892c0a9630 legacy=0x7f892c144da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:13.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 >> v1:172.21.15.32:6789/0 conn(0x7f892c14a450 legacy=0x7f892c135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 1 monclient: found mon.a 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.580+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c136700 con 0x7f892c149c10 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f89297fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f89297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:13.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f89297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:17:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 10 monclient: _renew_subs 2024-03-29T11:17:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892c1369b0 con 0x7f892c149c10 2024-03-29T11:17:13.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 10 monclient: _renew_subs 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.581+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f892c1deea0 con 0x7f892c149c10 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.582+0000 7f8932dfe640 1 librados: init done 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.583+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f891c001b70 con 0x7f892c149c10 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.584+0000 7f89297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.584+0000 7f892affd640 4 set_mon_vals no callback set 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.584+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f891c003a60 con 0x7f892c149c10 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 10 monclient: dump: 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:13.566 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f891c003c80 con 0x7f892c149c10 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f891c084bf0 con 0x7f892c149c10 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 10 client.4317.objecter ms_dispatch 0x7f892c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.585+0000 7f89297fa640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.586+0000 7f8930b9a640 10 monclient: get_auth_request con 0x7f891806dac0 auth_method 0 2024-03-29T11:17:13.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.586+0000 7f89297fa640 10 client.4317.objecter ms_handle_connect 0x7f891806dac0 2024-03-29T11:17:13.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.588+0000 7f8932dfe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:13.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.588+0000 7f8932dfe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:13.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.588+0000 7f8932dfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:13.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.588+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892c1357c0 con 0x7f892c149c10 2024-03-29T11:17:13.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.598+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f891c0505b0 con 0x7f892c149c10 2024-03-29T11:17:13.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.598+0000 7f89297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:13.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.598+0000 7f89297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:13.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.945+0000 7f8932dfe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:13.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.945+0000 7f8932dfe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:13.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.945+0000 7f8932dfe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:13.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.945+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f892c06a400 con 0x7f892c149c10 2024-03-29T11:17:13.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.946+0000 7f89297fa640 1 -- 172.21.15.3:0/1255087731 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f891c0524e0 con 0x7f892c149c10 2024-03-29T11:17:13.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.947+0000 7f89297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:13.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.947+0000 7f89297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:13.925 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:13.925 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 10 librados: watch_flush enter 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 10 librados: watch_flush exit 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 10 client.4317.objecter successfully canceled tick 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 >> v1:172.21.15.3:6816/844689376 conn(0x7f891806dac0 legacy=0x7f891806ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 10 monclient: shutdown 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.951+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 >> v1:172.21.15.3:6789/0 conn(0x7f892c149c10 legacy=0x7f892c139760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:13.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.952+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 shutdown_connections 2024-03-29T11:17:13.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.952+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 >> 172.21.15.3:0/1255087731 conn(0x7f892c0b3c10 msgr2=0x7f892c143510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:13.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.953+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 shutdown_connections 2024-03-29T11:17:13.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.953+0000 7f8932dfe640 1 -- 172.21.15.3:0/1255087731 wait complete. 2024-03-29T11:17:13.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:13.953+0000 7f8932dfe640 1 librados: shutdown 2024-03-29T11:17:13.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:20.756 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.925+0000 7fc8bf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.925+0000 7fc8bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.925+0000 7fc8bf59e640 10 monclient: monmap: 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:20.926466+0000 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:20.926466+0000 2024-03-29T11:17:20.903 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:20.904 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.904 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:20.904 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:20.904 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:20.904 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.928+0000 7fc8bf59e640 1 Processor -- start 2024-03-29T11:17:20.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.928+0000 7fc8bf59e640 1 -- start start 2024-03-29T11:17:20.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.928+0000 7fc8bf59e640 10 monclient: init 2024-03-29T11:17:20.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.930+0000 7fc8bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:20.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.930+0000 7fc8bf59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:20.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.930+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b8006410 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b802a640 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8be59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc8b8006410 0x7fc8b80c6ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:56577/0 (socket says 172.21.15.3:56577) 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bdd9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc8b802a640 0x7fc8b80320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39319/0 (socket says 172.21.15.3:39319) 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bdd9b640 1 -- 172.21.15.3:0/2008895395 learned_addr learned my addr 172.21.15.3:0/2008895395 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b80327d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b802c820 con 0x7fc8b802a640 2024-03-29T11:17:20.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b802a120 con 0x7fc8b8006410 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80c75b0 con 0x7fc8b80327d0 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bf59e640 10 monclient(hunting): authenticate will time out at 968.187829s 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677769868 0 0) 0x7fc8b802a120 con 0x7fc8b8006410 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.933+0000 7fc8bd59a640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8003520 con 0x7fc8b8006410 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782106830 0 0) 0x7fc8b802c820 con 0x7fc8b802a640 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:17:20.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b802a120 con 0x7fc8b802a640 2024-03-29T11:17:20.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2663570399 0 0) 0x7fc8a8003520 con 0x7fc8b8006410 2024-03-29T11:17:20.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.934+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 >> v1:172.21.15.3:6790/0 conn(0x7fc8b80327d0 legacy=0x7fc8b80d0350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:20.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 >> v1:172.21.15.3:6789/0 conn(0x7fc8b802a640 legacy=0x7fc8b80320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:20.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 1 monclient: found mon.noname-c 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b8033110 con 0x7fc8b8006410 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: _finish_auth 0 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc8b00015c0 con 0x7fc8b8006410 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bd59a640 10 monclient: dump: 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:20.916 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bf59e640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.936+0000 7fc8bf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8b00017e0 con 0x7fc8b8006410 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 1 -- 172.21.15.3:0/2008895395 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc8b0001b70 con 0x7fc8b8006410 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bd59a640 10 monclient: dump: 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.917 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bf59e640 4 set_mon_vals no callback set 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bf59e640 10 monclient: shutdown 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.937+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 >> v1:172.21.15.32:6789/0 conn(0x7fc8b8006410 legacy=0x7fc8b80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.938+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 shutdown_connections 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.938+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 >> 172.21.15.3:0/2008895395 conn(0x7fc8b80058f0 msgr2=0x7fc8b8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:20.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.938+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 shutdown_connections 2024-03-29T11:17:20.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.938+0000 7fc8bf59e640 1 -- 172.21.15.3:0/2008895395 wait complete. 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.939+0000 7fc8bf59e640 10 monclient: build_initial_monmap 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.939+0000 7fc8bf59e640 10 monclient: monmap: 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:20.920 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.939+0000 7fc8bf59e640 1 librados: starting msgr at 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 1 librados: starting objecter 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 1 Processor -- start 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 1 -- start start 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 1 librados: setting wanted keys 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 1 librados: calling monclient init 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.940+0000 7fc8bf59e640 10 monclient: init 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.941+0000 7fc8bf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.941+0000 7fc8bf59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-a con 0x7fc8b8006410 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-b con 0x7fc8b802a640 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 10 monclient(hunting): picked mon.noname-c con 0x7fc8b80327d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b80bfdc0 con 0x7fc8b8006410 2024-03-29T11:17:20.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b8152100 con 0x7fc8b802a640 2024-03-29T11:17:20.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b8152290 con 0x7fc8b80327d0 2024-03-29T11:17:20.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:20.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.943+0000 7fc8bf59e640 10 monclient(hunting): authenticate will time out at 968.197136s 2024-03-29T11:17:20.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.948+0000 7fc8bdd9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc8b802a640 0x7fc8b80baa30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:56661/0 (socket says 172.21.15.3:56661) 2024-03-29T11:17:20.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.948+0000 7fc8bdd9b640 1 -- 172.21.15.3:0/1372328652 learned_addr learned my addr 172.21.15.3:0/1372328652 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:20.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.948+0000 7fc8aeffd640 10 client.?.objecter ms_handle_connect 0x7fc8b802a640 2024-03-29T11:17:20.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.948+0000 7fc8aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:20.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 client.?.objecter ms_handle_connect 0x7fc8b8006410 2024-03-29T11:17:20.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:20.929 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3035673817 0 0) 0x7fc8b8152100 con 0x7fc8b802a640 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a0004eb0 con 0x7fc8b802a640 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155775224 0 0) 0x7fc8b80bfdc0 con 0x7fc8b8006410 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8b8152100 con 0x7fc8b8006410 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846040304 0 0) 0x7fc8a0004eb0 con 0x7fc8b802a640 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8b80bfdc0 con 0x7fc8b802a640 2024-03-29T11:17:20.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.950+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1065110449 0 0) 0x7fc8b8152100 con 0x7fc8b8006410 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8a0004eb0 con 0x7fc8b8006410 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc8b40019d0 con 0x7fc8b802a640 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): dump: 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:20.931 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc8b0001500 con 0x7fc8b8006410 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient(hunting): dump: 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:20.932 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 211836728 0 0) 0x7fc8a0004eb0 con 0x7fc8b8006410 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 >> v1:172.21.15.3:6790/0 conn(0x7fc8b80327d0 legacy=0x7fc8b80bb140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 >> v1:172.21.15.32:6789/0 conn(0x7fc8b802a640 legacy=0x7fc8b80baa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 monclient: found mon.a 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b814fd50 con 0x7fc8b8006410 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8b0000f50 con 0x7fc8b8006410 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc8b0004c30 con 0x7fc8b8006410 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:20.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8aeffd640 10 monclient: dump: 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 10 monclient: _renew_subs 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b80bf910 con 0x7fc8b8006410 2024-03-29T11:17:20.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bcd99640 4 set_mon_vals no callback set 2024-03-29T11:17:20.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:20.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 10 monclient: _renew_subs 2024-03-29T11:17:20.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:20.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b80c0bd0 con 0x7fc8b8006410 2024-03-29T11:17:20.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.951+0000 7fc8bf59e640 1 librados: init done 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fc8b001ac00 con 0x7fc8b8006410 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8bdd9b640 10 monclient: get_auth_request con 0x7fc8a00765d0 auth_method 0 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fc8b00849d0 con 0x7fc8b8006410 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 10 client.4323.objecter ms_dispatch 0x7fc8c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:20.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.954+0000 7fc8aeffd640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.955+0000 7fc8aeffd640 10 client.4323.objecter ms_handle_connect 0x7fc8a00765d0 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.957+0000 7fc8bf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.957+0000 7fc8bf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.957+0000 7fc8bf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:20.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.957+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b8004cd0 con 0x7fc8b8006410 2024-03-29T11:17:20.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.969+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc8b00502b0 con 0x7fc8b8006410 2024-03-29T11:17:20.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.970+0000 7fc8aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:20.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:20.970+0000 7fc8aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:21.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.290+0000 7fc8bf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8bf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8bf59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:21.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8b8037960 con 0x7fc8b8006410 2024-03-29T11:17:21.269 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:21.270 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:21.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8aeffd640 1 -- 172.21.15.3:0/1372328652 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fc8b00521e0 con 0x7fc8b8006410 2024-03-29T11:17:21.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:21.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.291+0000 7fc8aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.294+0000 7fc8bf59e640 10 librados: watch_flush enter 2024-03-29T11:17:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 10 librados: watch_flush exit 2024-03-29T11:17:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-29T11:17:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 10 client.4323.objecter successfully canceled tick 2024-03-29T11:17:21.273 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 >> v1:172.21.15.3:6816/844689376 conn(0x7fc8a00765d0 legacy=0x7fc8a0078a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 10 monclient: shutdown 2024-03-29T11:17:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 >> v1:172.21.15.3:6789/0 conn(0x7fc8b8006410 legacy=0x7fc8b80c02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 shutdown_connections 2024-03-29T11:17:21.274 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 >> 172.21.15.3:0/1372328652 conn(0x7fc8b80058f0 msgr2=0x7fc8b80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 shutdown_connections 2024-03-29T11:17:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.295+0000 7fc8bf59e640 1 -- 172.21.15.3:0/1372328652 wait complete. 2024-03-29T11:17:21.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:21.296+0000 7fc8bf59e640 1 librados: shutdown 2024-03-29T11:17:21.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:28.098 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.276+0000 7fae9279f640 10 monclient: get_monmap_and_config 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.276+0000 7fae9279f640 10 monclient: build_initial_monmap 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.276+0000 7fae9279f640 10 monclient: monmap: 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:28.277645+0000 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:28.277645+0000 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:28.254 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.277+0000 7fae9279f640 1 Processor -- start 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.278+0000 7fae9279f640 1 -- start start 2024-03-29T11:17:28.255 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.278+0000 7fae9279f640 10 monclient: init 2024-03-29T11:17:28.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:28.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:28.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c0affa0 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c0ae460 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c0ad690 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0adb40 con 0x7fae8c0affa0 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0aa510 con 0x7fae8c0ae460 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0ae120 con 0x7fae8c0ad690 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:28.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.279+0000 7fae9279f640 10 monclient(hunting): authenticate will time out at 975.533306s 2024-03-29T11:17:28.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.284+0000 7fae90f9c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae8c0ae460 0x7fae8c0acf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:55655/0 (socket says 172.21.15.3:55655) 2024-03-29T11:17:28.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.284+0000 7fae90f9c640 1 -- 172.21.15.3:0/1964049565 learned_addr learned my addr 172.21.15.3:0/1964049565 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208813154 0 0) 0x7fae8c0aa510 con 0x7fae8c0ae460 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae78003520 con 0x7fae8c0ae460 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894611198 0 0) 0x7fae8c0ae120 con 0x7fae8c0ad690 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:17:28.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.285+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae8c0aa510 con 0x7fae8c0ad690 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 516764756 0 0) 0x7fae78003520 con 0x7fae8c0ae460 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 >> v1:172.21.15.3:6790/0 conn(0x7fae8c0ad690 legacy=0x7fae8c149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 >> v1:172.21.15.3:6789/0 conn(0x7fae8c0affa0 legacy=0x7fae8c06b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 1 monclient: found mon.noname-c 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0a8840 con 0x7fae8c0ae460 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.288+0000 7fae83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae9279f640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:17:28.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae9279f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae7c001690 con 0x7fae8c0ae460 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: dump: 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae9279f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:28.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae7c0018b0 con 0x7fae8c0ae460 2024-03-29T11:17:28.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 1 -- 172.21.15.3:0/1964049565 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae7c001c40 con 0x7fae8c0ae460 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae83fff640 10 monclient: dump: 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:28.269 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae9279f640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.289+0000 7fae9279f640 4 set_mon_vals no callback set 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.290+0000 7fae9279f640 10 monclient: shutdown 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.290+0000 7fae9279f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.290+0000 7fae9279f640 1 -- 172.21.15.3:0/1964049565 >> v1:172.21.15.32:6789/0 conn(0x7fae8c0ae460 legacy=0x7fae8c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.290+0000 7fae9279f640 1 -- 172.21.15.3:0/1964049565 shutdown_connections 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.290+0000 7fae9279f640 1 -- 172.21.15.3:0/1964049565 >> 172.21.15.3:0/1964049565 conn(0x7fae8c0b3e80 msgr2=0x7fae8c0b4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.291+0000 7fae9279f640 1 -- 172.21.15.3:0/1964049565 shutdown_connections 2024-03-29T11:17:28.270 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.291+0000 7fae9279f640 1 -- 172.21.15.3:0/1964049565 wait complete. 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.292+0000 7fae9279f640 10 monclient: build_initial_monmap 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.292+0000 7fae9279f640 10 monclient: monmap: 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:28.271 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.292+0000 7fae9279f640 1 librados: starting msgr at 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.292+0000 7fae9279f640 1 librados: starting objecter 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.292+0000 7fae9279f640 1 Processor -- start 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 1 -- start start 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 1 librados: setting wanted keys 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 1 librados: calling monclient init 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 10 monclient: init 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:28.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.293+0000 7fae9279f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:28.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-b con 0x7fae8c0ad690 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:28.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-a con 0x7fae8c0ae460 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:28.275 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 10 monclient(hunting): picked mon.noname-c con 0x7fae8c0affa0 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c071a60 con 0x7fae8c0ae460 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c06e830 con 0x7fae8c0ad690 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c1c8690 con 0x7fae8c0affa0 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.296+0000 7fae9279f640 10 monclient(hunting): authenticate will time out at 975.550849s 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae9179d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fae8c0ad690 0x7fae8c071350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:55727/0 (socket says 172.21.15.3:55727) 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae9179d640 1 -- 172.21.15.3:0/3612800030 learned_addr learned my addr 172.21.15.3:0/3612800030 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 client.?.objecter ms_handle_connect 0x7fae8c0ad690 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135069240 0 0) 0x7fae8c06e830 con 0x7fae8c0ad690 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:28.276 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae70004eb0 con 0x7fae8c0ad690 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1907047883 0 0) 0x7fae70004eb0 con 0x7fae8c0ad690 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae8c06e830 con 0x7fae8c0ad690 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae84001b50 con 0x7fae8c0ad690 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.277 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.297+0000 7fae81ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.298+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3611733355 0 0) 0x7fae8c06e830 con 0x7fae8c0ad690 2024-03-29T11:17:28.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.298+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 >> v1:172.21.15.3:6790/0 conn(0x7fae8c0affa0 legacy=0x7fae8c1e6980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.299+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 >> v1:172.21.15.3:6789/0 conn(0x7fae8c0ae460 legacy=0x7fae8c1391a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 1 monclient: found mon.b 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c1c90f0 con 0x7fae8c0ad690 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:28.279 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 10 monclient: _renew_subs 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c13bb70 con 0x7fae8c0ad690 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 10 monclient: _renew_subs 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c1398b0 con 0x7fae8c0ad690 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae9279f640 1 librados: init done 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae84001db0 con 0x7fae8c0ad690 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae84003bf0 con 0x7fae8c0ad690 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:28.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae81ffb640 10 monclient: dump: 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:28.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.300+0000 7fae837fe640 4 set_mon_vals no callback set 2024-03-29T11:17:28.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.302+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fae84003d50 con 0x7fae8c0ad690 2024-03-29T11:17:28.282 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fae84084e10 con 0x7fae8c0ad690 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae81ffb640 10 client.4342.objecter ms_dispatch 0x7fae8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae81ffb640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae81ffb640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae81ffb640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.303+0000 7fae90f9c640 10 monclient: get_auth_request con 0x7fae7006db30 auth_method 0 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.304+0000 7fae81ffb640 10 client.4342.objecter ms_handle_connect 0x7fae7006db30 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.304+0000 7fae9279f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.304+0000 7fae9279f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.304+0000 7fae9279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.283 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.304+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6c005080 con 0x7fae8c0ad690 2024-03-29T11:17:28.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.315+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae84050520 con 0x7fae8c0ad690 2024-03-29T11:17:28.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.315+0000 7fae81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:28.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.315+0000 7fae81ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:28.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.649+0000 7fae9279f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:28.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.649+0000 7fae9279f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:28.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.649+0000 7fae9279f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:28.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.649+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae6c005330 con 0x7fae8c0ad690 2024-03-29T11:17:28.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.653+0000 7fae81ffb640 1 -- 172.21.15.3:0/3612800030 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fae84052620 con 0x7fae8c0ad690 2024-03-29T11:17:28.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.653+0000 7fae81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:28.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.653+0000 7fae81ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:28.632 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:28.632 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.656+0000 7fae9279f640 10 librados: watch_flush enter 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.657+0000 7fae9279f640 10 librados: watch_flush exit 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.657+0000 7fae9279f640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.657+0000 7fae9279f640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.658+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 >> v1:172.21.15.3:6816/844689376 conn(0x7fae7006db30 legacy=0x7fae7006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.658+0000 7fae9279f640 10 monclient: shutdown 2024-03-29T11:17:28.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.658+0000 7fae9279f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:28.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.658+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 >> v1:172.21.15.32:6789/0 conn(0x7fae8c0ad690 legacy=0x7fae8c071350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:28.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.661+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 shutdown_connections 2024-03-29T11:17:28.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.661+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 >> 172.21.15.3:0/3612800030 conn(0x7fae8c0b3e80 msgr2=0x7fae8c0ac790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:28.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.662+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 shutdown_connections 2024-03-29T11:17:28.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.662+0000 7fae9279f640 1 -- 172.21.15.3:0/3612800030 wait complete. 2024-03-29T11:17:28.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:28.662+0000 7fae9279f640 1 librados: shutdown 2024-03-29T11:17:28.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:35.465 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.640+0000 7f8436194640 10 monclient: get_monmap_and_config 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.640+0000 7f8436194640 10 monclient: build_initial_monmap 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.640+0000 7f8436194640 10 monclient: monmap: 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:35.617 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:35.641032+0000 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:35.641032+0000 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.642+0000 7f8436194640 1 Processor -- start 2024-03-29T11:17:35.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.642+0000 7f8436194640 1 -- start start 2024-03-29T11:17:35.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.642+0000 7f8436194640 10 monclient: init 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient(hunting): picked mon.noname-b con 0x7f8428006a70 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient(hunting): picked mon.noname-a con 0x7f842802b100 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient(hunting): picked mon.noname-c con 0x7f8428029c30 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:35.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842802a270 con 0x7f842802b100 2024-03-29T11:17:35.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f842802c440 con 0x7f8428029c30 2024-03-29T11:17:35.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280c9440 con 0x7f8428006a70 2024-03-29T11:17:35.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:35.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.643+0000 7f8436194640 10 monclient(hunting): authenticate will time out at 982.897460s 2024-03-29T11:17:35.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.646+0000 7f8435192640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8428006a70 0x7f84280c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42529/0 (socket says 172.21.15.3:42529) 2024-03-29T11:17:35.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.646+0000 7f8435192640 1 -- 172.21.15.3:0/2411092043 learned_addr learned my addr 172.21.15.3:0/2411092043 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703515140 0 0) 0x7f842802a270 con 0x7f842802b100 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.648+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f841c003520 con 0x7f842802b100 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.649+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2175141488 0 0) 0x7f841c003520 con 0x7f842802b100 2024-03-29T11:17:35.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.649+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 >> v1:172.21.15.3:6790/0 conn(0x7f8428006a70 legacy=0x7f84280c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 >> v1:172.21.15.32:6789/0 conn(0x7f8428029c30 legacy=0x7f84280d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 1 monclient: found mon.noname-a 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:17:35.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280c95d0 con 0x7f842802b100 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f843004f730 con 0x7f842802b100 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.650+0000 7f8427fff640 10 monclient: dump: 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:35.629 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:35.630 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:35.630 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.653+0000 7f8436194640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:17:35.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.653+0000 7f8436194640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:35.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.654+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8430075f80 con 0x7f842802b100 2024-03-29T11:17:35.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.654+0000 7f8427fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:35.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.654+0000 7f8436194640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:35.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.654+0000 7f8436194640 4 set_mon_vals no callback set 2024-03-29T11:17:35.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8427fff640 1 -- 172.21.15.3:0/2411092043 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f843004f080 con 0x7f842802b100 2024-03-29T11:17:35.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8427fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:35.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8427fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8427fff640 10 monclient: dump: 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8436194640 10 monclient: shutdown 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8436194640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:35.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.655+0000 7f8436194640 1 -- 172.21.15.3:0/2411092043 >> v1:172.21.15.3:6789/0 conn(0x7f842802b100 legacy=0x7f84280cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:35.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.656+0000 7f8436194640 1 -- 172.21.15.3:0/2411092043 shutdown_connections 2024-03-29T11:17:35.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.656+0000 7f8436194640 1 -- 172.21.15.3:0/2411092043 >> 172.21.15.3:0/2411092043 conn(0x7f84280058f0 msgr2=0x7f8428005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:35.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.659+0000 7f8436194640 1 -- 172.21.15.3:0/2411092043 shutdown_connections 2024-03-29T11:17:35.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.660+0000 7f8436194640 1 -- 172.21.15.3:0/2411092043 wait complete. 2024-03-29T11:17:35.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 10 monclient: build_initial_monmap 2024-03-29T11:17:35.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 10 monclient: monmap: 2024-03-29T11:17:35.639 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 librados: starting msgr at 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 librados: starting objecter 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 Processor -- start 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 -- start start 2024-03-29T11:17:35.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 librados: setting wanted keys 2024-03-29T11:17:35.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 1 librados: calling monclient init 2024-03-29T11:17:35.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.661+0000 7f8436194640 10 monclient: init 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient(hunting): picked mon.noname-b con 0x7f8428006a70 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient(hunting): picked mon.noname-a con 0x7f8428029c30 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient(hunting): picked mon.noname-c con 0x7f84280ccc30 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:35.644 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280cc7a0 con 0x7f8428029c30 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280cc450 con 0x7f8428006a70 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84280cdcc0 con 0x7f84280ccc30 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.662+0000 7f8436194640 10 monclient(hunting): authenticate will time out at 982.916917s 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.666+0000 7f8435993640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f84280ccc30 0x7f8428185370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42617/0 (socket says 172.21.15.3:42617) 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.666+0000 7f8435993640 1 -- 172.21.15.3:0/3562280406 learned_addr learned my addr 172.21.15.3:0/3562280406 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.667+0000 7f8425ffb640 10 client.?.objecter ms_handle_connect 0x7f84280ccc30 2024-03-29T11:17:35.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.667+0000 7f8425ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624716620 0 0) 0x7f84280cdcc0 con 0x7f84280ccc30 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8414004eb0 con 0x7f84280ccc30 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4078683208 0 0) 0x7f8414004eb0 con 0x7f84280ccc30 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84280cdcc0 con 0x7f84280ccc30 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f842c0018e0 con 0x7f84280ccc30 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.668+0000 7f8425ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.669+0000 7f8425ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.669+0000 7f8425ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:35.647 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.648 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.669+0000 7f8425ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:35.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.672+0000 7f8425ffb640 10 client.?.objecter ms_handle_connect 0x7f8428029c30 2024-03-29T11:17:35.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.672+0000 7f8425ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:35.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 client.?.objecter ms_handle_connect 0x7f8428006a70 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 406638804 0 0) 0x7f84280cdcc0 con 0x7f84280ccc30 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 >> v1:172.21.15.32:6789/0 conn(0x7f8428006a70 legacy=0x7f84280c7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 >> v1:172.21.15.3:6789/0 conn(0x7f8428029c30 legacy=0x7f84280ce3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 1 monclient: found mon.c 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84280cdfa0 con 0x7f84280ccc30 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:17:35.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 10 monclient: _renew_subs 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84280cead0 con 0x7f84280ccc30 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 10 monclient: _renew_subs 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84280cbfe0 con 0x7f84280ccc30 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.673+0000 7f8436194640 1 librados: init done 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.674+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f842c001a40 con 0x7f84280ccc30 2024-03-29T11:17:35.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.674+0000 7f8425ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:35.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.674+0000 7f84277fe640 4 set_mon_vals no callback set 2024-03-29T11:17:35.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.677+0000 7f8436194640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.678+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f842c003890 con 0x7f84280ccc30 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.678+0000 7f8425ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.678+0000 7f8425ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.678+0000 7f8425ffb640 10 monclient: dump: 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:35.656 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:35.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.678+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f842c003af0 con 0x7f84280ccc30 2024-03-29T11:17:35.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.679+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f842c084aa0 con 0x7f84280ccc30 2024-03-29T11:17:35.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.679+0000 7f8425ffb640 10 client.4340.objecter ms_dispatch 0x7f8430000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:35.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.679+0000 7f8425ffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:35.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.679+0000 7f8425ffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:35.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.679+0000 7f8425ffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:17:35.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.680+0000 7f8435192640 10 monclient: get_auth_request con 0x7f841406dac0 auth_method 0 2024-03-29T11:17:35.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.680+0000 7f8436194640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:35.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.680+0000 7f8436194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:35.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.680+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8410005080 con 0x7f84280ccc30 2024-03-29T11:17:35.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.680+0000 7f8425ffb640 10 client.4340.objecter ms_handle_connect 0x7f841406dac0 2024-03-29T11:17:35.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.688+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f842c050400 con 0x7f84280ccc30 2024-03-29T11:17:35.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.688+0000 7f8425ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:35.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:35.688+0000 7f8425ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:35.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.019+0000 7f8436194640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.020+0000 7f8436194640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.020+0000 7f8436194640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.020+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8410005330 con 0x7f84280ccc30 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.022+0000 7f8425ffb640 1 -- 172.21.15.3:0/3562280406 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f842c052330 con 0x7f84280ccc30 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.022+0000 7f8425ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:35.999 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.022+0000 7f8425ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:36.000 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:36.000 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:36.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 10 librados: watch_flush enter 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 10 librados: watch_flush exit 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 >> v1:172.21.15.3:6816/844689376 conn(0x7f841406dac0 legacy=0x7f841406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 10 monclient: shutdown 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 >> v1:172.21.15.3:6790/0 conn(0x7f84280ccc30 legacy=0x7f8428185370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:36.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 shutdown_connections 2024-03-29T11:17:36.004 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.026+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 >> 172.21.15.3:0/3562280406 conn(0x7f84280058f0 msgr2=0x7f8428160050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:36.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.028+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 shutdown_connections 2024-03-29T11:17:36.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.028+0000 7f8436194640 1 -- 172.21.15.3:0/3562280406 wait complete. 2024-03-29T11:17:36.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:36.028+0000 7f8436194640 1 librados: shutdown 2024-03-29T11:17:36.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:42.833 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.014+0000 7f75bfe33640 10 monclient: get_monmap_and_config 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.014+0000 7f75bfe33640 10 monclient: build_initial_monmap 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.014+0000 7f75bfe33640 10 monclient: monmap: 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:43.015152+0000 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:43.015152+0000 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:42.993 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.015+0000 7f75bfe33640 1 Processor -- start 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.016+0000 7f75bfe33640 1 -- start start 2024-03-29T11:17:42.994 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.016+0000 7f75bfe33640 10 monclient: init 2024-03-29T11:17:42.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.017+0000 7f75bfe33640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:42.995 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.017+0000 7f75bfe33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.017+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80a9d00 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-a con 0x7f75b80aaef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80af110 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8141e60 con 0x7f75b80aaef0 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b80b0090 con 0x7f75b80af110 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8141a30 con 0x7f75b80a9d00 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:42.997 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.019+0000 7f75bfe33640 10 monclient(hunting): authenticate will time out at 990.273526s 2024-03-29T11:17:42.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.021+0000 7f75bdbcf640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f75b80a9d00 0x7f75b80af980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55073/0 (socket says 172.21.15.3:55073) 2024-03-29T11:17:42.998 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.021+0000 7f75bdbcf640 1 -- 172.21.15.3:0/2698878122 learned_addr learned my addr 172.21.15.3:0/2698878122 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:43.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.022+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468384048 0 0) 0x7f75b8141a30 con 0x7f75b80a9d00 2024-03-29T11:17:43.002 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.022+0000 7f75bcbcd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.022+0000 7f75bcbcd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.022+0000 7f75bcbcd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.022+0000 7f75bcbcd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:17:43.003 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.023+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a4003520 con 0x7f75b80a9d00 2024-03-29T11:17:43.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 383558543 0 0) 0x7f75a4003520 con 0x7f75b80a9d00 2024-03-29T11:17:43.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 >> v1:172.21.15.32:6789/0 conn(0x7f75b80af110 legacy=0x7f75b81498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 >> v1:172.21.15.3:6789/0 conn(0x7f75b80aaef0 legacy=0x7f75b80aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:43.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.005 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 monclient: found mon.noname-b 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b81422f0 con 0x7f75b80a9d00 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75b4001690 con 0x7f75b80a9d00 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: dump: 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:43.006 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b4001920 con 0x7f75b80a9d00 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 1 -- 172.21.15.3:0/2698878122 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75b4001cb0 con 0x7f75b80a9d00 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bcbcd640 10 monclient: dump: 2024-03-29T11:17:43.007 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bfe33640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bfe33640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.026+0000 7f75bfe33640 4 set_mon_vals no callback set 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.028+0000 7f75bfe33640 10 monclient: shutdown 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.028+0000 7f75bfe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:43.008 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.028+0000 7f75bfe33640 1 -- 172.21.15.3:0/2698878122 >> v1:172.21.15.3:6790/0 conn(0x7f75b80a9d00 legacy=0x7f75b80af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.028+0000 7f75bfe33640 1 -- 172.21.15.3:0/2698878122 shutdown_connections 2024-03-29T11:17:43.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.028+0000 7f75bfe33640 1 -- 172.21.15.3:0/2698878122 >> 172.21.15.3:0/2698878122 conn(0x7f75b80b3c10 msgr2=0x7f75b80b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:43.009 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.032+0000 7f75bfe33640 1 -- 172.21.15.3:0/2698878122 shutdown_connections 2024-03-29T11:17:43.010 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.033+0000 7f75bfe33640 1 -- 172.21.15.3:0/2698878122 wait complete. 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.034+0000 7f75bfe33640 10 monclient: build_initial_monmap 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.034+0000 7f75bfe33640 10 monclient: monmap: 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:43.011 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.012 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 librados: starting msgr at 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 librados: starting objecter 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 Processor -- start 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 -- start start 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 librados: setting wanted keys 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 1 librados: calling monclient init 2024-03-29T11:17:43.013 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.035+0000 7f75bfe33640 10 monclient: init 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.036+0000 7f75bfe33640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.036+0000 7f75bfe33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80a9d00 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-a con 0x7f75b80aaef0 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80af110 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8131ad0 con 0x7f75b80aaef0 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8131db0 con 0x7f75b80af110 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8132090 con 0x7f75b80a9d00 2024-03-29T11:17:43.016 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:43.017 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.037+0000 7f75bfe33640 10 monclient(hunting): authenticate will time out at 990.291200s 2024-03-29T11:17:43.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75bd3ce640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f75b80aaef0 0x7f75b806fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55207/0 (socket says 172.21.15.3:55207) 2024-03-29T11:17:43.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75bd3ce640 1 -- 172.21.15.3:0/4019385726 learned_addr learned my addr 172.21.15.3:0/4019385726 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:43.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter ms_handle_connect 0x7f75b80af110 2024-03-29T11:17:43.023 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter ms_handle_connect 0x7f75b80aaef0 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter ms_handle_connect 0x7f75b80a9d00 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280911525 0 0) 0x7f75b8131db0 con 0x7f75b80af110 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f759c004eb0 con 0x7f75b80af110 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.043+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845960926 0 0) 0x7f75b8132090 con 0x7f75b80a9d00 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8131db0 con 0x7f75b80a9d00 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018235069 0 0) 0x7f75b8131ad0 con 0x7f75b80aaef0 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8132090 con 0x7f75b80aaef0 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 772066003 0 0) 0x7f75b8132090 con 0x7f75b80aaef0 2024-03-29T11:17:43.025 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75b8131ad0 con 0x7f75b80aaef0 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4172820566 0 0) 0x7f75b8131db0 con 0x7f75b80a9d00 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75b8132090 con 0x7f75b80a9d00 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75a8003e10 con 0x7f75b80aaef0 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:43.026 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75b40013b0 con 0x7f75b80a9d00 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient(hunting): dump: 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.027 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3123087604 0 0) 0x7f75b8131ad0 con 0x7f75b80aaef0 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 >> v1:172.21.15.3:6790/0 conn(0x7f75b80a9d00 legacy=0x7f75b806f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 >> v1:172.21.15.32:6789/0 conn(0x7f75b80af110 legacy=0x7f75b81df940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 monclient: found mon.a 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.044+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8071cc0 con 0x7f75b80aaef0 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75ae7fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:43.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 5 monclient: authenticate success, global_id 4335 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 10 monclient: _renew_subs 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b8130ec0 con 0x7f75b80aaef0 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 10 monclient: _renew_subs 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b8072730 con 0x7f75b80aaef0 2024-03-29T11:17:43.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.045+0000 7f75bfe33640 1 librados: init done 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.049+0000 7f75bfe33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.049+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75a8003f70 con 0x7f75b80aaef0 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.049+0000 7f75ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.049+0000 7f75af7fe640 4 set_mon_vals no callback set 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.050+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75a8000ed0 con 0x7f75b80aaef0 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75ae7fc640 10 monclient: dump: 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:43.031 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75bfe33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75bfe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b8130d30 con 0x7f75b80aaef0 2024-03-29T11:17:43.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.051+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f75a80010f0 con 0x7f75b80aaef0 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f75a8084740 con 0x7f75b80aaef0 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75ae7fc640 10 client.4335.objecter ms_dispatch 0x7f75b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75ae7fc640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75ae7fc640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75ae7fc640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.055+0000 7f75bdbcf640 10 monclient: get_auth_request con 0x7f759c06e0b0 auth_method 0 2024-03-29T11:17:43.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.056+0000 7f75ae7fc640 10 client.4335.objecter ms_handle_connect 0x7f759c06e0b0 2024-03-29T11:17:43.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.063+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75a80500e0 con 0x7f75b80aaef0 2024-03-29T11:17:43.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.063+0000 7f75ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:43.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.063+0000 7f75ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:43.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.398+0000 7f75bfe33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.399+0000 7f75bfe33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.399+0000 7f75bfe33640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:17:43.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.399+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75b806a400 con 0x7f75b80aaef0 2024-03-29T11:17:43.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.402+0000 7f75ae7fc640 1 -- 172.21.15.3:0/4019385726 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f75a8052010 con 0x7f75b80aaef0 2024-03-29T11:17:43.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.402+0000 7f75ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:43.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.402+0000 7f75ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:43.383 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:43.383 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:43.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 10 librados: watch_flush enter 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 10 librados: watch_flush exit 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 10 client.4335.objecter successfully canceled tick 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 >> v1:172.21.15.3:6816/844689376 conn(0x7f759c06e0b0 legacy=0x7f759c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 10 monclient: shutdown 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.409+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 >> v1:172.21.15.3:6789/0 conn(0x7f75b80aaef0 legacy=0x7f75b806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.410+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 shutdown_connections 2024-03-29T11:17:43.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.410+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 >> 172.21.15.3:0/4019385726 conn(0x7f75b80b3c10 msgr2=0x7f75b8069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:43.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.413+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 shutdown_connections 2024-03-29T11:17:43.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.413+0000 7f75bfe33640 1 -- 172.21.15.3:0/4019385726 wait complete. 2024-03-29T11:17:43.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:43.414+0000 7f75bfe33640 1 librados: shutdown 2024-03-29T11:17:43.408 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:50.215 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.390+0000 7f3f9b59e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.390+0000 7f3f9b59e640 10 monclient: build_initial_monmap 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.390+0000 7f3f9b59e640 10 monclient: monmap: 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:50.391161+0000 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:50.391161+0000 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:50.367 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.391+0000 7f3f9b59e640 1 Processor -- start 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.391+0000 7f3f9b59e640 1 -- start start 2024-03-29T11:17:50.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.391+0000 7f3f9b59e640 10 monclient: init 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f9c072bf0 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f9c0b0570 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f9c0afd70 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:50.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c071090 con 0x7f3f9c0b0570 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c06ffa0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c13be70 con 0x7f3f9c0afd70 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9b59e640 10 monclient(hunting): authenticate will time out at 997.646694s 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9a59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3f9c072bf0 0x7f3f9c0743e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:37726/0 (socket says 172.21.15.3:37726) 2024-03-29T11:17:50.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.392+0000 7f3f9a59c640 1 -- 172.21.15.3:0/1550968571 learned_addr learned my addr 172.21.15.3:0/1550968571 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:50.372 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303516395 0 0) 0x7f3f9c06ffa0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f88003520 con 0x7f3f9c072bf0 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165882095 0 0) 0x7f3f9c13be70 con 0x7f3f9c0afd70 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.393+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c06ffa0 con 0x7f3f9c0afd70 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337992237 0 0) 0x7f3f88003520 con 0x7f3f9c072bf0 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 >> v1:172.21.15.3:6790/0 conn(0x7f3f9c0afd70 legacy=0x7f3f9c13b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.373 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 >> v1:172.21.15.3:6789/0 conn(0x7f3f9c0b0570 legacy=0x7f3f9c06f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 monclient: found mon.noname-c 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c13c000 con 0x7f3f9c072bf0 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: _finish_auth 0 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9b59e640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3f94001a10 con 0x7f3f9c072bf0 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.374 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9959a640 10 monclient: dump: 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:50.375 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.394+0000 7f3f9b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f94001c60 con 0x7f3f9c072bf0 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9b59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9b59e640 4 set_mon_vals no callback set 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 1 -- 172.21.15.3:0/1550968571 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3f94001ff0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9959a640 10 monclient: dump: 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.376 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9b59e640 10 monclient: shutdown 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.395+0000 7f3f9b59e640 1 -- 172.21.15.3:0/1550968571 >> v1:172.21.15.32:6789/0 conn(0x7f3f9c072bf0 legacy=0x7f3f9c0743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.396+0000 7f3f9b59e640 1 -- 172.21.15.3:0/1550968571 shutdown_connections 2024-03-29T11:17:50.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.396+0000 7f3f9b59e640 1 -- 172.21.15.3:0/1550968571 >> 172.21.15.3:0/1550968571 conn(0x7f3f9c0b0c80 msgr2=0x7f3f9c0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.396+0000 7f3f9b59e640 1 -- 172.21.15.3:0/1550968571 shutdown_connections 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.396+0000 7f3f9b59e640 1 -- 172.21.15.3:0/1550968571 wait complete. 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.397+0000 7f3f9b59e640 10 monclient: build_initial_monmap 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.397+0000 7f3f9b59e640 10 monclient: monmap: 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:50.380 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 librados: starting msgr at 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 librados: starting objecter 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 Processor -- start 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 -- start start 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 librados: setting wanted keys 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 1 librados: calling monclient init 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.398+0000 7f3f9b59e640 10 monclient: init 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:50.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3f9c072bf0 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3f9c0afd70 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3f9c0b0570 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c14d740 con 0x7f3f9c0afd70 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c150110 con 0x7f3f9c072bf0 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f9c1dff10 con 0x7f3f9c0b0570 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9b59e640 10 monclient(hunting): authenticate will time out at 997.653429s 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9a59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3f9c072bf0 0x7f3f9c14fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:38152/0 (socket says 172.21.15.3:38152) 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f9a59c640 1 -- 172.21.15.3:0/4082704174 learned_addr learned my addr 172.21.15.3:0/4082704174 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 client.?.objecter ms_handle_connect 0x7f3f9c072bf0 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:50.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 client.?.objecter ms_handle_connect 0x7f3f9c0afd70 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251763734 0 0) 0x7f3f9c150110 con 0x7f3f9c072bf0 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.399+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c004eb0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.400+0000 7f3f92ffd640 10 client.?.objecter ms_handle_connect 0x7f3f9c0b0570 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.400+0000 7f3f92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482131106 0 0) 0x7f3f9c1dff10 con 0x7f3f9c0b0570 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:17:50.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f9c150110 con 0x7f3f9c0b0570 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821704684 0 0) 0x7f3f7c004eb0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f9c1dff10 con 0x7f3f9c072bf0 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3f94001f70 con 0x7f3f9c072bf0 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.384 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056702030 0 0) 0x7f3f9c14d740 con 0x7f3f9c0afd70 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.401+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f7c004eb0 con 0x7f3f9c0afd70 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085393222 0 0) 0x7f3f9c1dff10 con 0x7f3f9c072bf0 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 >> v1:172.21.15.3:6790/0 conn(0x7f3f9c0b0570 legacy=0x7f3f9c1df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 >> v1:172.21.15.3:6789/0 conn(0x7f3f9c0afd70 legacy=0x7f3f9c1df0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 1 monclient: found mon.b 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f9c14f020 con 0x7f3f9c072bf0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 10 monclient: _renew_subs 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f9c14f300 con 0x7f3f9c072bf0 2024-03-29T11:17:50.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 10 monclient: _renew_subs 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9c14cea0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.402+0000 7f3f9b59e640 1 librados: init done 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f94001570 con 0x7f3f9c072bf0 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3f94004fb0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f92ffd640 10 monclient: dump: 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.387 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:50.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.403+0000 7f3f98d99640 4 set_mon_vals no callback set 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.404+0000 7f3f9b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.404+0000 7f3f9b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.404+0000 7f3f9b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.404+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80005080 con 0x7f3f9c072bf0 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.405+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f3f9401af80 con 0x7f3f9c072bf0 2024-03-29T11:17:50.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f3f94084e90 con 0x7f3f9c072bf0 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f92ffd640 10 client.4354.objecter ms_dispatch 0x7f3f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f92ffd640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f92ffd640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f92ffd640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.406+0000 7f3f99d9b640 10 monclient: get_auth_request con 0x7f3f7c06de00 auth_method 0 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.407+0000 7f3f92ffd640 10 client.4354.objecter ms_handle_connect 0x7f3f7c06de00 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.413+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f940505a0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.413+0000 7f3f92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:50.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.413+0000 7f3f92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:50.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.788+0000 7f3f9b59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:50.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.788+0000 7f3f9b59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:50.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.788+0000 7f3f9b59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:17:50.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.788+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f80005330 con 0x7f3f9c072bf0 2024-03-29T11:17:50.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.789+0000 7f3f92ffd640 1 -- 172.21.15.3:0/4082704174 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f3f940526a0 con 0x7f3f9c072bf0 2024-03-29T11:17:50.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.789+0000 7f3f92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:50.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.789+0000 7f3f92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:50.766 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:50.766 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 10 librados: watch_flush enter 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 10 librados: watch_flush exit 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 >> v1:172.21.15.3:6816/844689376 conn(0x7f3f7c06de00 legacy=0x7f3f7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.792+0000 7f3f9b59e640 10 monclient: shutdown 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 >> v1:172.21.15.32:6789/0 conn(0x7f3f9c072bf0 legacy=0x7f3f9c14fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:50.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 shutdown_connections 2024-03-29T11:17:50.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 >> 172.21.15.3:0/4082704174 conn(0x7f3f9c0b0c80 msgr2=0x7f3f9c069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:50.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 shutdown_connections 2024-03-29T11:17:50.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 -- 172.21.15.3:0/4082704174 wait complete. 2024-03-29T11:17:50.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:50.793+0000 7f3f9b59e640 1 librados: shutdown 2024-03-29T11:17:50.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:57.591 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:57.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 10 monclient: get_monmap_and_config 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 10 monclient: build_initial_monmap 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 10 monclient: monmap: 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:17:57.766025+0000 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:17:57.766025+0000 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:17:57.745 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 1 Processor -- start 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 1 -- start start 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.764+0000 7fd754802640 10 monclient: init 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.765+0000 7fd754802640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.765+0000 7fd754802640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.765+0000 7fd754802640 10 monclient(hunting): picked mon.noname-c con 0x7fd75014f8d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.765+0000 7fd754802640 10 monclient(hunting): picked mon.noname-a con 0x7fd7500aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 10 monclient(hunting): picked mon.noname-b con 0x7fd75008b370 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7500aed20 con 0x7fd7500aab00 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75014f3b0 con 0x7fd75014f8d0 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75015d570 con 0x7fd75008b370 2024-03-29T11:17:57.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:57.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.766+0000 7fd754802640 10 monclient(hunting): authenticate will time out at 1005.021030s 2024-03-29T11:17:57.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.769+0000 7fd74e59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd75014f8d0 0x7fd7500a9630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:43145/0 (socket says 172.21.15.3:43145) 2024-03-29T11:17:57.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.769+0000 7fd74e59c640 1 -- 172.21.15.3:0/2040387085 learned_addr learned my addr 172.21.15.3:0/2040387085 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865949209 0 0) 0x7fd75014f3b0 con 0x7fd75014f8d0 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd738003520 con 0x7fd75014f8d0 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905228831 0 0) 0x7fd7500aed20 con 0x7fd7500aab00 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:57.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd75014f3b0 con 0x7fd7500aab00 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909339572 0 0) 0x7fd75015d570 con 0x7fd75008b370 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7500aed20 con 0x7fd75008b370 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417572946 0 0) 0x7fd738003520 con 0x7fd75014f8d0 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 >> v1:172.21.15.3:6790/0 conn(0x7fd75008b370 legacy=0x7fd75008b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 >> v1:172.21.15.3:6789/0 conn(0x7fd7500aab00 legacy=0x7fd7500ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 monclient: found mon.noname-c 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75015e710 con 0x7fd75014f8d0 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: _finish_auth 0 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd748005010 con 0x7fd75014f8d0 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd74d59a640 10 monclient: dump: 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:57.752 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd754802640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.770+0000 7fd754802640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd748005170 con 0x7fd75014f8d0 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd754802640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd754802640 4 set_mon_vals no callback set 2024-03-29T11:17:57.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 1 -- 172.21.15.3:0/2040387085 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd748001790 con 0x7fd75014f8d0 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.771+0000 7fd74d59a640 10 monclient: dump: 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:57.754 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.772+0000 7fd754802640 10 monclient: shutdown 2024-03-29T11:17:57.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.772+0000 7fd754802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.772+0000 7fd754802640 1 -- 172.21.15.3:0/2040387085 >> v1:172.21.15.32:6789/0 conn(0x7fd75014f8d0 legacy=0x7fd7500a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.775+0000 7fd754802640 1 -- 172.21.15.3:0/2040387085 shutdown_connections 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.775+0000 7fd754802640 1 -- 172.21.15.3:0/2040387085 >> 172.21.15.3:0/2040387085 conn(0x7fd7500b3c10 msgr2=0x7fd7500b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.775+0000 7fd754802640 1 -- 172.21.15.3:0/2040387085 shutdown_connections 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.775+0000 7fd754802640 1 -- 172.21.15.3:0/2040387085 wait complete. 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.776+0000 7fd754802640 10 monclient: build_initial_monmap 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.776+0000 7fd754802640 10 monclient: monmap: 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:57.757 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 librados: starting msgr at 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 librados: starting objecter 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 Processor -- start 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 -- start start 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 librados: setting wanted keys 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 1 librados: calling monclient init 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.777+0000 7fd754802640 10 monclient: init 2024-03-29T11:17:57.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.778+0000 7fd754802640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:57.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.778+0000 7fd754802640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:57.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.778+0000 7fd754802640 10 monclient(hunting): picked mon.noname-c con 0x7fd75008b370 addr v1:172.21.15.3:6790/0 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.778+0000 7fd754802640 10 monclient(hunting): picked mon.noname-b con 0x7fd7500aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 10 monclient(hunting): picked mon.noname-a con 0x7fd75014f8d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75007d1f0 con 0x7fd75014f8d0 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75007d420 con 0x7fd7500aab00 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75007d700 con 0x7fd75008b370 2024-03-29T11:17:57.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.780+0000 7fd754802640 10 monclient(hunting): authenticate will time out at 1005.035225s 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.781+0000 7fd74dd9b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd7500aab00 0x7fd750070ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:43267/0 (socket says 172.21.15.3:43267) 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.781+0000 7fd74dd9b640 1 -- 172.21.15.3:0/3070750902 learned_addr learned my addr 172.21.15.3:0/3070750902 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.779+0000 7fd74e59c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fd75008b370 0x7fd75014b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:60362/0 (socket says 172.21.15.3:60362) 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.781+0000 7fd73effd640 10 client.?.objecter ms_handle_connect 0x7fd7500aab00 2024-03-29T11:17:57.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.781+0000 7fd73effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 client.?.objecter ms_handle_connect 0x7fd75008b370 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182763591 0 0) 0x7fd75007d700 con 0x7fd75008b370 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.783+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd730004eb0 con 0x7fd75008b370 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127267155 0 0) 0x7fd75007d420 con 0x7fd7500aab00 2024-03-29T11:17:57.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7300051d0 con 0x7fd7500aab00 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1913659031 0 0) 0x7fd730004eb0 con 0x7fd75008b370 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd75007d420 con 0x7fd75008b370 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd7480019c0 con 0x7fd75008b370 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): dump: 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.765 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 client.?.objecter ms_handle_connect 0x7fd75014f8d0 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 775262841 0 0) 0x7fd75007d420 con 0x7fd75008b370 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 >> v1:172.21.15.32:6789/0 conn(0x7fd7500aab00 legacy=0x7fd750070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 >> v1:172.21.15.3:6789/0 conn(0x7fd75014f8d0 legacy=0x7fd7500711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:57.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 monclient: found mon.c 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75007d9e0 con 0x7fd75008b370 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient: _finish_auth 0 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd73effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd754802640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd754802640 10 monclient: _renew_subs 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd754802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.784+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75014be40 con 0x7fd75008b370 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd754802640 10 client.4358.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd754802640 10 monclient: _renew_subs 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd754802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:57.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7501efa60 con 0x7fd75008b370 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd754802640 1 librados: init done 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd748004f50 con 0x7fd75008b370 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fd748001ff0 con 0x7fd75008b370 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd73effd640 10 monclient: dump: 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:57.768 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:17:57.769 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:17:57.769 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:17:57.769 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:17:57.769 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:17:57.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.785+0000 7fd74cd99640 4 set_mon_vals no callback set 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.788+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fd74801b150 con 0x7fd75008b370 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.788+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fd748084c60 con 0x7fd75008b370 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.788+0000 7fd73effd640 10 client.4358.objecter ms_dispatch 0x7fd750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.788+0000 7fd73effd640 3 client.4358.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd73effd640 3 client.4358.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:57.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd73effd640 20 client.4358.objecter dump_active .. 0 homeless 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd74dd9b640 10 monclient: get_auth_request con 0x7fd73006e060 auth_method 0 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd754802640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd754802640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd754802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd734005080 con 0x7fd75008b370 2024-03-29T11:17:57.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.789+0000 7fd73effd640 10 client.4358.objecter ms_handle_connect 0x7fd73006e060 2024-03-29T11:17:57.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.798+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd748050680 con 0x7fd75008b370 2024-03-29T11:17:57.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.798+0000 7fd73effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:57.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:57.798+0000 7fd73effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:58.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.130+0000 7fd754802640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.130+0000 7fd754802640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.130+0000 7fd754802640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:17:58.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.130+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd734005330 con 0x7fd75008b370 2024-03-29T11:17:58.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.131+0000 7fd73effd640 1 -- 172.21.15.3:0/3070750902 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fd748050680 con 0x7fd75008b370 2024-03-29T11:17:58.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.131+0000 7fd73effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:58.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.131+0000 7fd73effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:58.109 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:17:58.110 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:58.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.134+0000 7fd754802640 10 librados: watch_flush enter 2024-03-29T11:17:58.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.134+0000 7fd754802640 10 librados: watch_flush exit 2024-03-29T11:17:58.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.134+0000 7fd754802640 20 client.4358.objecter shutdown clearing up homeless session... 2024-03-29T11:17:58.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.134+0000 7fd754802640 10 client.4358.objecter successfully canceled tick 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 >> v1:172.21.15.3:6816/844689376 conn(0x7fd73006e060 legacy=0x7fd730070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 10 monclient: shutdown 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 >> v1:172.21.15.3:6790/0 conn(0x7fd75008b370 legacy=0x7fd75014b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 shutdown_connections 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 >> 172.21.15.3:0/3070750902 conn(0x7fd7500b3c10 msgr2=0x7fd75008aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 shutdown_connections 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 -- 172.21.15.3:0/3070750902 wait complete. 2024-03-29T11:17:58.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:17:58.135+0000 7fd754802640 1 librados: shutdown 2024-03-29T11:17:58.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:04.937 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.115+0000 7fdffdff4640 10 monclient: get_monmap_and_config 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.115+0000 7fdffdff4640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.115+0000 7fdffdff4640 10 monclient: monmap: 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:05.117165+0000 2024-03-29T11:18:05.093 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:05.117165+0000 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.116+0000 7fdffdff4640 1 Processor -- start 2024-03-29T11:18:05.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.116+0000 7fdffdff4640 1 -- start start 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.116+0000 7fdffdff4640 10 monclient: init 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-c con 0x7fdff80a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-b con 0x7fdff80aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-a con 0x7fdff8146f80 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff80aed20 con 0x7fdff8146f80 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff80aa8b0 con 0x7fdff80a98a0 2024-03-29T11:18:05.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8146a60 con 0x7fdff80aab00 2024-03-29T11:18:05.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdffdff4640 10 monclient(hunting): authenticate will time out at 1012.372569s 2024-03-29T11:18:05.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdff6ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdff80aab00 0x7fdff81437e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42920/0 (socket says 172.21.15.3:42920) 2024-03-29T11:18:05.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.117+0000 7fdff6ffd640 1 -- 172.21.15.3:0/2229946450 learned_addr learned my addr 172.21.15.3:0/2229946450 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:05.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593479836 0 0) 0x7fdff8146a60 con 0x7fdff80aab00 2024-03-29T11:18:05.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe0003520 con 0x7fdff80aab00 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958270488 0 0) 0x7fdff80aed20 con 0x7fdff8146f80 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.118+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8146a60 con 0x7fdff8146f80 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631745823 0 0) 0x7fdfe0003520 con 0x7fdff80aab00 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 >> v1:172.21.15.32:6789/0 conn(0x7fdff80a98a0 legacy=0x7fdff80ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 >> v1:172.21.15.3:6789/0 conn(0x7fdff8146f80 legacy=0x7fdff8151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 monclient: found mon.noname-b 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff8143ef0 con 0x7fdff80aab00 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdfe8001690 con 0x7fdff80aab00 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 10 monclient: dump: 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.099 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdffdff4640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:18:05.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdffdff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:05.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.119+0000 7fdff67fc640 1 -- 172.21.15.3:0/2229946450 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfe80018b0 con 0x7fdff80aab00 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdffdff4640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdffdff4640 4 set_mon_vals no callback set 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdffdff4640 10 monclient: shutdown 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdffdff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.120+0000 7fdffdff4640 1 -- 172.21.15.3:0/2229946450 >> v1:172.21.15.3:6790/0 conn(0x7fdff80aab00 legacy=0x7fdff81437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.121+0000 7fdffdff4640 1 -- 172.21.15.3:0/2229946450 shutdown_connections 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.121+0000 7fdffdff4640 1 -- 172.21.15.3:0/2229946450 >> 172.21.15.3:0/2229946450 conn(0x7fdff80b3c10 msgr2=0x7fdff80b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.121+0000 7fdffdff4640 1 -- 172.21.15.3:0/2229946450 shutdown_connections 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.121+0000 7fdffdff4640 1 -- 172.21.15.3:0/2229946450 wait complete. 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 10 monclient: build_initial_monmap 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 10 monclient: monmap: 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:05.103 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 librados: starting msgr at 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 librados: starting objecter 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 Processor -- start 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 -- start start 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 librados: setting wanted keys 2024-03-29T11:18:05.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 1 librados: calling monclient init 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.122+0000 7fdffdff4640 10 monclient: init 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.123+0000 7fdffdff4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.123+0000 7fdffdff4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.123+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-a con 0x7fdff80a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-c con 0x7fdff80aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdff6ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdff80aab00 0x7fdff81429e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:43124/0 (socket says 172.21.15.3:43124) 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdff6ffd640 1 -- 172.21.15.3:0/1112959202 learned_addr learned my addr 172.21.15.3:0/1112959202 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdfdffff640 10 client.?.objecter ms_handle_connect 0x7fdff80aab00 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdfdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 10 monclient(hunting): picked mon.noname-b con 0x7fdff8146f80 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8136f30 con 0x7fdff80a98a0 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8135d80 con 0x7fdff8146f80 2024-03-29T11:18:05.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdfdffff640 10 client.?.objecter ms_handle_connect 0x7fdff8146f80 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdfdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdff8135f10 con 0x7fdff80aab00 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.124+0000 7fdffdff4640 10 monclient(hunting): authenticate will time out at 1012.379915s 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 client.?.objecter ms_handle_connect 0x7fdff80a98a0 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653157025 0 0) 0x7fdff8135f10 con 0x7fdff80aab00 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:18:05.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfe4004eb0 con 0x7fdff80aab00 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675948099 0 0) 0x7fdff8136f30 con 0x7fdff80a98a0 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8135f10 con 0x7fdff80a98a0 2024-03-29T11:18:05.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589940828 0 0) 0x7fdff8135d80 con 0x7fdff8146f80 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff8136f30 con 0x7fdff8146f80 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780269149 0 0) 0x7fdfe4004eb0 con 0x7fdff80aab00 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff8135d80 con 0x7fdff80aab00 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdfe80047e0 con 0x7fdff80aab00 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): dump: 2024-03-29T11:18:05.110 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.125+0000 7fdfdffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146634148 0 0) 0x7fdff8136f30 con 0x7fdff8146f80 2024-03-29T11:18:05.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfe4004eb0 con 0x7fdff8146f80 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdff0001cb0 con 0x7fdff8146f80 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient(hunting): dump: 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:05.112 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3303853142 0 0) 0x7fdff8135f10 con 0x7fdff80a98a0 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdff8136f30 con 0x7fdff80a98a0 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1473813850 0 0) 0x7fdff8135d80 con 0x7fdff80aab00 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 >> v1:172.21.15.32:6789/0 conn(0x7fdff8146f80 legacy=0x7fdff81db7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 >> v1:172.21.15.3:6789/0 conn(0x7fdff80a98a0 legacy=0x7fdff8135670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 monclient: found mon.c 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff81430f0 con 0x7fdff80aab00 2024-03-29T11:18:05.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdfdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdffdff4640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdffdff4640 10 monclient: _renew_subs 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdffdff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdff8134d90 con 0x7fdff80aab00 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.126+0000 7fdffdff4640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdffdff4640 10 monclient: _renew_subs 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdffdff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdff81dbf00 con 0x7fdff80aab00 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdffdff4640 1 librados: init done 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfe8001aa0 con 0x7fdff80aab00 2024-03-29T11:18:05.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdfe8004d90 con 0x7fdff80aab00 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdfdffff640 10 monclient: dump: 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:05.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.127+0000 7fdff5ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.128+0000 7fdffdff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.128+0000 7fdffdff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.128+0000 7fdffdff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.128+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd8005080 con 0x7fdff80aab00 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.129+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fdfe8004ef0 con 0x7fdff80aab00 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fdfe8084a50 con 0x7fdff80aab00 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdfdffff640 10 client.4364.objecter ms_dispatch 0x7fdff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdfdffff640 3 client.4364.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdfdffff640 3 client.4364.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdfdffff640 20 client.4364.objecter dump_active .. 0 homeless 2024-03-29T11:18:05.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.130+0000 7fdff77fe640 10 monclient: get_auth_request con 0x7fdfe406e020 auth_method 0 2024-03-29T11:18:05.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.131+0000 7fdfdffff640 10 client.4364.objecter ms_handle_connect 0x7fdfe406e020 2024-03-29T11:18:05.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.138+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfe8050330 con 0x7fdff80aab00 2024-03-29T11:18:05.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.138+0000 7fdfdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:05.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.138+0000 7fdfdffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:05.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.524+0000 7fdffdff4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.524+0000 7fdffdff4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.524+0000 7fdffdff4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.524+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfd8005330 con 0x7fdff80aab00 2024-03-29T11:18:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.650+0000 7fdfdffff640 1 -- 172.21.15.3:0/1112959202 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fdfe8052260 con 0x7fdff80aab00 2024-03-29T11:18:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.650+0000 7fdfdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.650+0000 7fdfdffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:05.628 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:05.628 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.653+0000 7fdffdff4640 10 librados: watch_flush enter 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 10 librados: watch_flush exit 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 20 client.4364.objecter shutdown clearing up homeless session... 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 10 client.4364.objecter successfully canceled tick 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 >> v1:172.21.15.3:6816/844689376 conn(0x7fdfe406e020 legacy=0x7fdfe40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 10 monclient: shutdown 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 >> v1:172.21.15.3:6790/0 conn(0x7fdff80aab00 legacy=0x7fdff81429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 shutdown_connections 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 >> 172.21.15.3:0/1112959202 conn(0x7fdff80b3c10 msgr2=0x7fdff80ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 shutdown_connections 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 -- 172.21.15.3:0/1112959202 wait complete. 2024-03-29T11:18:05.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:05.654+0000 7fdffdff4640 1 librados: shutdown 2024-03-29T11:18:05.647 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:12.455 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:12.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.630+0000 7ffbeba06640 10 monclient: get_monmap_and_config 2024-03-29T11:18:12.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.630+0000 7ffbeba06640 10 monclient: build_initial_monmap 2024-03-29T11:18:12.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.630+0000 7ffbeba06640 10 monclient: monmap: 2024-03-29T11:18:12.608 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:12.608 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:12.631420+0000 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:12.631420+0000 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:12.609 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:12.610 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:12.610 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.631+0000 7ffbeba06640 1 Processor -- start 2024-03-29T11:18:12.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 1 -- start start 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient: init 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe40b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:12.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe40b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.632+0000 7ffbeba06640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40aab60 con 0x7ffbe40b3560 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbeba06640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40aae40 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbeba06640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe40ab120 con 0x7ffbe40b28d0 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbeba06640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbeba06640 10 monclient(hunting): authenticate will time out at 1019.887016s 2024-03-29T11:18:12.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbe8fa1640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffbe40b2cb0 0x7ffbe4142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:37318/0 (socket says 172.21.15.3:37318) 2024-03-29T11:18:12.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.633+0000 7ffbe8fa1640 1 -- 172.21.15.3:0/1449407070 learned_addr learned my addr 172.21.15.3:0/1449407070 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925150365 0 0) 0x7ffbe40aae40 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:18:12.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc003520 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1261461822 0 0) 0x7ffbcc003520 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.635+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 >> v1:172.21.15.3:6790/0 conn(0x7ffbe40b28d0 legacy=0x7ffbe413bc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.636+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 >> v1:172.21.15.3:6789/0 conn(0x7ffbe40b3560 legacy=0x7ffbe41428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.637+0000 7ffbe3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.637+0000 7ffbe3fff640 1 monclient: found mon.noname-c 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe413c380 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbeba06640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbeba06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffbdc001690 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.620 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: dump: 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:12.621 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbeba06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 1 -- 172.21.15.3:0/1449407070 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbdc001920 con 0x7ffbe40b2cb0 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbe3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbeba06640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.638+0000 7ffbeba06640 4 set_mon_vals no callback set 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.639+0000 7ffbeba06640 10 monclient: shutdown 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.639+0000 7ffbeba06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.639+0000 7ffbeba06640 1 -- 172.21.15.3:0/1449407070 >> v1:172.21.15.32:6789/0 conn(0x7ffbe40b2cb0 legacy=0x7ffbe4142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.640+0000 7ffbeba06640 1 -- 172.21.15.3:0/1449407070 shutdown_connections 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.640+0000 7ffbeba06640 1 -- 172.21.15.3:0/1449407070 >> 172.21.15.3:0/1449407070 conn(0x7ffbe4066a00 msgr2=0x7ffbe4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.640+0000 7ffbeba06640 1 -- 172.21.15.3:0/1449407070 shutdown_connections 2024-03-29T11:18:12.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.640+0000 7ffbeba06640 1 -- 172.21.15.3:0/1449407070 wait complete. 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.641+0000 7ffbeba06640 10 monclient: build_initial_monmap 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.641+0000 7ffbeba06640 10 monclient: monmap: 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:12.623 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 librados: starting msgr at 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 librados: starting objecter 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 Processor -- start 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 -- start start 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 librados: setting wanted keys 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 1 librados: calling monclient init 2024-03-29T11:18:12.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.642+0000 7ffbeba06640 10 monclient: init 2024-03-29T11:18:12.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-c con 0x7ffbe40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-a con 0x7ffbe413bcd0 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient(hunting): picked mon.noname-b con 0x7ffbe41336d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe413b8b0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe413a710 con 0x7ffbe41336d0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe4139250 con 0x7ffbe40b28d0 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:12.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.643+0000 7ffbeba06640 10 monclient(hunting): authenticate will time out at 1019.897863s 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.644+0000 7ffbe9fa3640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ffbe41336d0 0x7ffbe41e2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:38524/0 (socket says 172.21.15.3:38524) 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.644+0000 7ffbe9fa3640 1 -- 172.21.15.3:0/1522838563 learned_addr learned my addr 172.21.15.3:0/1522838563 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.645+0000 7ffbe1ffb640 10 client.?.objecter ms_handle_connect 0x7ffbe41336d0 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.645+0000 7ffbe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543569921 0 0) 0x7ffbe413a710 con 0x7ffbe41336d0 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd0004eb0 con 0x7ffbe41336d0 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 client.?.objecter ms_handle_connect 0x7ffbe413bcd0 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.646+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 894857025 0 0) 0x7ffbd0004eb0 con 0x7ffbe41336d0 2024-03-29T11:18:12.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbe413a710 con 0x7ffbe41336d0 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffbd8001750 con 0x7ffbe41336d0 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36185155 0 0) 0x7ffbe413b8b0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd0004eb0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1745435062 0 0) 0x7ffbd0004eb0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.647+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbe413b8b0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 client.?.objecter ms_handle_connect 0x7ffbe40b28d0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffbdc0047e0 con 0x7ffbe413bcd0 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:12.633 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1516533723 0 0) 0x7ffbe413a710 con 0x7ffbe41336d0 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 >> v1:172.21.15.3:6790/0 conn(0x7ffbe40b28d0 legacy=0x7ffbe41371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 >> v1:172.21.15.3:6789/0 conn(0x7ffbe413bcd0 legacy=0x7ffbe4132fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.648+0000 7ffbe1ffb640 1 monclient: found mon.b 2024-03-29T11:18:12.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe41393e0 con 0x7ffbe41336d0 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbd8001ce0 con 0x7ffbe41336d0 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffbd80018b0 con 0x7ffbe41336d0 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.649+0000 7ffbe1ffb640 10 monclient: dump: 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:12.635 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 10 monclient: _renew_subs 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe413f3b0 con 0x7ffbe41336d0 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:12.636 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 10 monclient: _renew_subs 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe413a0c0 con 0x7ffbe41336d0 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbeba06640 1 librados: init done 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.650+0000 7ffbe37fe640 4 set_mon_vals no callback set 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.652+0000 7ffbeba06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.652+0000 7ffbeba06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.652+0000 7ffbeba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.652+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc4005080 con 0x7ffbe41336d0 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.653+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ffbd801a840 con 0x7ffbe41336d0 2024-03-29T11:18:12.637 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7ffbd8084910 con 0x7ffbe41336d0 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe1ffb640 10 client.4369.objecter ms_dispatch 0x7ffbe4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe1ffb640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe1ffb640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe1ffb640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.654+0000 7ffbe97a2640 10 monclient: get_auth_request con 0x7ffbd006dcd0 auth_method 0 2024-03-29T11:18:12.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.655+0000 7ffbe1ffb640 10 client.4369.objecter ms_handle_connect 0x7ffbd006dcd0 2024-03-29T11:18:12.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.663+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbd8050020 con 0x7ffbe41336d0 2024-03-29T11:18:12.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.663+0000 7ffbe1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:12.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.663+0000 7ffbe1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.993+0000 7ffbeba06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.993+0000 7ffbeba06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.993+0000 7ffbeba06640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.993+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbc4005330 con 0x7ffbe41336d0 2024-03-29T11:18:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.995+0000 7ffbe1ffb640 1 -- 172.21.15.3:0/1522838563 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7ffbd8052120 con 0x7ffbe41336d0 2024-03-29T11:18:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.995+0000 7ffbe1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.995+0000 7ffbe1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:12.973 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:12.973 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:12.999+0000 7ffbeba06640 10 librados: watch_flush enter 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 10 librados: watch_flush exit 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 10 client.4369.objecter successfully canceled tick 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 >> v1:172.21.15.3:6816/844689376 conn(0x7ffbd006dcd0 legacy=0x7ffbd0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 10 monclient: shutdown 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 >> v1:172.21.15.32:6789/0 conn(0x7ffbe41336d0 legacy=0x7ffbe41e2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:12.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 shutdown_connections 2024-03-29T11:18:12.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 >> 172.21.15.3:0/1522838563 conn(0x7ffbe4066a00 msgr2=0x7ffbe4140c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:12.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.000+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 shutdown_connections 2024-03-29T11:18:12.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.001+0000 7ffbeba06640 1 -- 172.21.15.3:0/1522838563 wait complete. 2024-03-29T11:18:12.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:13.001+0000 7ffbeba06640 1 librados: shutdown 2024-03-29T11:18:12.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:19.804 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.978+0000 7f27b1f35640 10 monclient: get_monmap_and_config 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.979+0000 7f27b1f35640 10 monclient: build_initial_monmap 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.979+0000 7f27b1f35640 10 monclient: monmap: 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:19.979916+0000 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:19.979916+0000 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.956 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:19.957 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:19.957 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:19.957 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.980+0000 7f27b1f35640 1 Processor -- start 2024-03-29T11:18:19.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.980+0000 7f27b1f35640 1 -- start start 2024-03-29T11:18:19.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.980+0000 7f27b1f35640 10 monclient: init 2024-03-29T11:18:19.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:19.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-a con 0x7f27a40389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-c con 0x7f27a4038dc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-b con 0x7f27a4030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a40c54c0 con 0x7f27a40389e0 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4036430 con 0x7f27a4038dc0 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a4039840 con 0x7f27a4030660 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.981+0000 7f27b1f35640 10 monclient(hunting): authenticate will time out at 1027.235498s 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.982+0000 7f27abfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27a4038dc0 0x7f27a40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:56346/0 (socket says 172.21.15.3:56346) 2024-03-29T11:18:19.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.982+0000 7f27abfff640 1 -- 172.21.15.3:0/3354645877 learned_addr learned my addr 172.21.15.3:0/3354645877 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:19.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703553862 0 0) 0x7f27a4036430 con 0x7f27a4038dc0 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f279c003520 con 0x7f27a4038dc0 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342121409 0 0) 0x7f27a40c54c0 con 0x7f27a40389e0 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a4036430 con 0x7f27a40389e0 2024-03-29T11:18:19.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2929966065 0 0) 0x7f27a4036430 con 0x7f27a40389e0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 >> v1:172.21.15.3:6790/0 conn(0x7f27a4030660 legacy=0x7f27a40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 >> v1:172.21.15.32:6789/0 conn(0x7f27a4038dc0 legacy=0x7f27a40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 monclient: found mon.noname-a 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a4035690 con 0x7f27a40389e0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f27a0001670 con 0x7f27a40389e0 2024-03-29T11:18:19.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: dump: 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.967 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a00017d0 con 0x7f27a40389e0 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 1 -- 172.21.15.3:0/3354645877 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f27a0001be0 con 0x7f27a40389e0 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27ab7fe640 10 monclient: dump: 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.968 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27b1f35640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27b1f35640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.986+0000 7f27b1f35640 4 set_mon_vals no callback set 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.987+0000 7f27b1f35640 10 monclient: shutdown 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.987+0000 7f27b1f35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:19.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.987+0000 7f27b1f35640 1 -- 172.21.15.3:0/3354645877 >> v1:172.21.15.3:6789/0 conn(0x7f27a40389e0 legacy=0x7f27a4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.987+0000 7f27b1f35640 1 -- 172.21.15.3:0/3354645877 shutdown_connections 2024-03-29T11:18:19.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.988+0000 7f27b1f35640 1 -- 172.21.15.3:0/3354645877 >> 172.21.15.3:0/3354645877 conn(0x7f27a4005c00 msgr2=0x7f27a4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:19.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.988+0000 7f27b1f35640 1 -- 172.21.15.3:0/3354645877 shutdown_connections 2024-03-29T11:18:19.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.988+0000 7f27b1f35640 1 -- 172.21.15.3:0/3354645877 wait complete. 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 10 monclient: build_initial_monmap 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 10 monclient: monmap: 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:19.974 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 librados: starting msgr at 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 librados: starting objecter 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 Processor -- start 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 -- start start 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 librados: setting wanted keys 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 1 librados: calling monclient init 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.989+0000 7f27b1f35640 10 monclient: init 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:19.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-c con 0x7f27a4030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-b con 0x7f27a40389e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient(hunting): picked mon.noname-a con 0x7f27a400c760 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a40c0ee0 con 0x7f27a400c760 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a40c2440 con 0x7f27a40389e0 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a400d040 con 0x7f27a4030660 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.990+0000 7f27b1f35640 10 monclient(hunting): authenticate will time out at 1027.244874s 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27abfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f27a40389e0 0x7f27a40ca230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:56560/0 (socket says 172.21.15.3:56560) 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27abfff640 1 -- 172.21.15.3:0/873478178 learned_addr learned my addr 172.21.15.3:0/873478178 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27a97fa640 10 client.?.objecter ms_handle_connect 0x7f27a40389e0 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27a97fa640 10 client.?.objecter ms_handle_connect 0x7f27a4030660 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.993+0000 7f27a97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414859683 0 0) 0x7f27a400d040 con 0x7f27a4030660 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:19.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2790004eb0 con 0x7f27a4030660 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752551686 0 0) 0x7f27a40c2440 con 0x7f27a40389e0 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a400d040 con 0x7f27a40389e0 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027692069 0 0) 0x7f2790004eb0 con 0x7f27a4030660 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27a40c2440 con 0x7f27a4030660 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f27a0001b20 con 0x7f27a4030660 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:19.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): dump: 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.978 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2045818504 0 0) 0x7f27a40c2440 con 0x7f27a4030660 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 >> v1:172.21.15.32:6789/0 conn(0x7f27a40389e0 legacy=0x7f27a40ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.994+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 >> v1:172.21.15.3:6789/0 conn(0x7f27a400c760 legacy=0x7f27a4163b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 1 monclient: found mon.c 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a400d1d0 con 0x7f27a4030660 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 10 monclient: _finish_auth 0 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.996+0000 7f27a97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:18:19.979 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:19.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:18:19.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 10 monclient: _renew_subs 2024-03-29T11:18:19.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a40c5650 con 0x7f27a4030660 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 10 monclient: _renew_subs 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a4158e40 con 0x7f27a4030660 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.997+0000 7f27b1f35640 1 librados: init done 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.998+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a0001c80 con 0x7f27a4030660 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.998+0000 7f27a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.998+0000 7f27b1f35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.998+0000 7f27aaffd640 4 set_mon_vals no callback set 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.999+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f27a0003a40 con 0x7f27a4030660 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.999+0000 7f27a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:19.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.999+0000 7f27a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:19.999+0000 7f27a97fa640 10 monclient: dump: 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.000+0000 7f27b1f35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.000+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f27a0003ba0 con 0x7f27a4030660 2024-03-29T11:18:19.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27b1f35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4007b90 con 0x7f27a4030660 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f27a0083940 con 0x7f27a4030660 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27a97fa640 10 client.4373.objecter ms_dispatch 0x7f27ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27a97fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27a97fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.001+0000 7f27a97fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.003+0000 7f27abfff640 10 monclient: get_auth_request con 0x7f279006dd00 auth_method 0 2024-03-29T11:18:19.984 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.004+0000 7f27a97fa640 10 client.4373.objecter ms_handle_connect 0x7f279006dd00 2024-03-29T11:18:19.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.010+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27a004f220 con 0x7f27a4030660 2024-03-29T11:18:19.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.010+0000 7f27a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:19.987 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.010+0000 7f27a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:20.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.335+0000 7f27b1f35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.335+0000 7f27b1f35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.335+0000 7f27b1f35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:20.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.335+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27a400cbe0 con 0x7f27a4030660 2024-03-29T11:18:20.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.336+0000 7f27a97fa640 1 -- 172.21.15.3:0/873478178 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f27a400cbe0 con 0x7f27a4030660 2024-03-29T11:18:20.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.336+0000 7f27a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:20.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.336+0000 7f27a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:20.314 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:20.314 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 10 librados: watch_flush enter 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 10 librados: watch_flush exit 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 >> v1:172.21.15.3:6816/844689376 conn(0x7f279006dd00 legacy=0x7f27900701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 10 monclient: shutdown 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 >> v1:172.21.15.3:6790/0 conn(0x7f27a4030660 legacy=0x7f27a4158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 shutdown_connections 2024-03-29T11:18:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.340+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 >> 172.21.15.3:0/873478178 conn(0x7f27a4005c00 msgr2=0x7f27a40c27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.341+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 shutdown_connections 2024-03-29T11:18:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.341+0000 7f27b1f35640 1 -- 172.21.15.3:0/873478178 wait complete. 2024-03-29T11:18:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:20.343+0000 7f27b1f35640 1 librados: shutdown 2024-03-29T11:18:20.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:27.144 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:27.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.327+0000 7f94f6f0e640 10 monclient: get_monmap_and_config 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.327+0000 7f94f6f0e640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.327+0000 7f94f6f0e640 10 monclient: monmap: 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:27.328137+0000 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:27.328137+0000 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:27.306 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.329+0000 7f94f6f0e640 1 Processor -- start 2024-03-29T11:18:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.329+0000 7f94f6f0e640 1 -- start start 2024-03-29T11:18:27.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.329+0000 7f94f6f0e640 10 monclient: init 2024-03-29T11:18:27.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f94f00b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f94f00b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f94f00b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00aab60 con 0x7f94f00b3560 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00aae40 con 0x7f94f00b28d0 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f00ab120 con 0x7f94f00b2cb0 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.330+0000 7f94f6f0e640 10 monclient(hunting): authenticate will time out at 1034.584773s 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.334+0000 7f94f4caa640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f94f00b28d0 0x7f94f013bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:59557/0 (socket says 172.21.15.3:59557) 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.334+0000 7f94f4caa640 1 -- 172.21.15.3:0/2678402894 learned_addr learned my addr 172.21.15.3:0/2678402894 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377700316 0 0) 0x7f94f00ab120 con 0x7f94f00b2cb0 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d8003520 con 0x7f94f00b2cb0 2024-03-29T11:18:27.312 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4121136715 0 0) 0x7f94d8003520 con 0x7f94f00b2cb0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 >> v1:172.21.15.32:6789/0 conn(0x7f94f00b28d0 legacy=0x7f94f013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 >> v1:172.21.15.3:6789/0 conn(0x7f94f00b3560 legacy=0x7f94f01428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 monclient: found mon.noname-b 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f013c380 con 0x7f94f00b2cb0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f94e4001690 con 0x7f94f00b2cb0 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: dump: 2024-03-29T11:18:27.313 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94f6f0e640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:18:27.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94f6f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:27.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e4001860 con 0x7f94f00b2cb0 2024-03-29T11:18:27.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2678402894 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f94e4001cb0 con 0x7f94f00b2cb0 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.335+0000 7f94ef7fe640 10 monclient: dump: 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.338+0000 7f94f6f0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:27.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.338+0000 7f94f6f0e640 4 set_mon_vals no callback set 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 10 monclient: shutdown 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 1 -- 172.21.15.3:0/2678402894 >> v1:172.21.15.3:6790/0 conn(0x7f94f00b2cb0 legacy=0x7f94f0142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 1 -- 172.21.15.3:0/2678402894 shutdown_connections 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 1 -- 172.21.15.3:0/2678402894 >> 172.21.15.3:0/2678402894 conn(0x7f94f0066a00 msgr2=0x7f94f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 1 -- 172.21.15.3:0/2678402894 shutdown_connections 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.339+0000 7f94f6f0e640 1 -- 172.21.15.3:0/2678402894 wait complete. 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.340+0000 7f94f6f0e640 10 monclient: build_initial_monmap 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.340+0000 7f94f6f0e640 10 monclient: monmap: 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:27.318 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 librados: starting msgr at 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 librados: starting objecter 2024-03-29T11:18:27.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 Processor -- start 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 -- start start 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 librados: setting wanted keys 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 1 librados: calling monclient init 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.341+0000 7f94f6f0e640 10 monclient: init 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.342+0000 7f94f6f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.342+0000 7f94f6f0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:27.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.342+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f94f00b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:27.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f94f013e4b0 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:27.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f94f013e930 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f0132790 con 0x7f94f013e4b0 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f013eda0 con 0x7f94f00b28d0 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f01dfb80 con 0x7f94f013e930 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.345+0000 7f94f6f0e640 10 monclient(hunting): authenticate will time out at 1034.599635s 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.346+0000 7f94effff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f94f013e4b0 0x7f94f01ded60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37551/0 (socket says 172.21.15.3:37551) 2024-03-29T11:18:27.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.346+0000 7f94effff640 1 -- 172.21.15.3:0/810290434 learned_addr learned my addr 172.21.15.3:0/810290434 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 client.?.objecter ms_handle_connect 0x7f94f013e4b0 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596119867 0 0) 0x7f94f0132790 con 0x7f94f013e4b0 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:18:27.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.348+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94dc004eb0 con 0x7f94f013e4b0 2024-03-29T11:18:27.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 client.?.objecter ms_handle_connect 0x7f94f013e930 2024-03-29T11:18:27.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293454603 0 0) 0x7f94f01dfb80 con 0x7f94f013e930 2024-03-29T11:18:27.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:18:27.329 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.351+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f0132790 con 0x7f94f013e930 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1133906043 0 0) 0x7f94dc004eb0 con 0x7f94f013e4b0 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94f01dfb80 con 0x7f94f013e4b0 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f94e4001c30 con 0x7f94f013e4b0 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): dump: 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.330 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1842390423 0 0) 0x7f94f0132790 con 0x7f94f013e930 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94dc004eb0 con 0x7f94f013e930 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f94e0001690 con 0x7f94f013e930 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.352+0000 7f94ed7fa640 10 monclient(hunting): dump: 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:27.331 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:27.332 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.353+0000 7f94ed7fa640 10 client.?.objecter ms_handle_connect 0x7f94f00b28d0 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.353+0000 7f94ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2369497855 0 0) 0x7f94dc004eb0 con 0x7f94f013e930 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 >> v1:172.21.15.32:6789/0 conn(0x7f94f00b28d0 legacy=0x7f94f0145ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 >> v1:172.21.15.3:6789/0 conn(0x7f94f013e4b0 legacy=0x7f94f01ded60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 monclient: found mon.c 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f01329c0 con 0x7f94f013e930 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: _finish_auth 0 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:18:27.334 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94e0001ba0 con 0x7f94f013e930 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 10 monclient: _renew_subs 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f013f740 con 0x7f94f013e930 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f94e0003790 con 0x7f94f013e930 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94eeffd640 4 set_mon_vals no callback set 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94ed7fa640 10 monclient: dump: 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:27.335 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 10 monclient: _renew_subs 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f006eb20 con 0x7f94f013e930 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.354+0000 7f94f6f0e640 1 librados: init done 2024-03-29T11:18:27.336 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.357+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f94e0019760 con 0x7f94f013e930 2024-03-29T11:18:27.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.357+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f94e00835a0 con 0x7f94f013e930 2024-03-29T11:18:27.337 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.358+0000 7f94f4caa640 10 monclient: get_auth_request con 0x7f94dc06de90 auth_method 0 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.358+0000 7f94ed7fa640 10 client.4379.objecter ms_dispatch 0x7f94f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.358+0000 7f94ed7fa640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.358+0000 7f94ed7fa640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.358+0000 7f94ed7fa640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.357+0000 7f94f6f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.359+0000 7f94f6f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.359+0000 7f94f6f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.359+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f006f150 con 0x7f94f013e930 2024-03-29T11:18:27.338 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.359+0000 7f94ed7fa640 10 client.4379.objecter ms_handle_connect 0x7f94dc06de90 2024-03-29T11:18:27.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.367+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94e004ee80 con 0x7f94f013e930 2024-03-29T11:18:27.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.367+0000 7f94ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:27.344 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.367+0000 7f94ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:27.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.699+0000 7f94f6f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.699+0000 7f94f6f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.699+0000 7f94f6f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:27.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.699+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94f006de80 con 0x7f94f013e930 2024-03-29T11:18:27.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.700+0000 7f94ed7fa640 1 -- 172.21.15.3:0/810290434 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f94e0050db0 con 0x7f94f013e930 2024-03-29T11:18:27.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.700+0000 7f94ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:27.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.700+0000 7f94ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:27.677 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:27.677 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:27.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.703+0000 7f94f6f0e640 10 librados: watch_flush enter 2024-03-29T11:18:27.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.703+0000 7f94f6f0e640 10 librados: watch_flush exit 2024-03-29T11:18:27.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.703+0000 7f94f6f0e640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:18:27.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.703+0000 7f94f6f0e640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:18:27.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.705+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 >> v1:172.21.15.3:6816/844689376 conn(0x7f94dc06de90 legacy=0x7f94dc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.706+0000 7f94f6f0e640 10 monclient: shutdown 2024-03-29T11:18:27.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.706+0000 7f94f6f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:27.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.706+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 >> v1:172.21.15.3:6790/0 conn(0x7f94f013e930 legacy=0x7f94f01df470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:27.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.706+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 shutdown_connections 2024-03-29T11:18:27.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.706+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 >> 172.21.15.3:0/810290434 conn(0x7f94f0066a00 msgr2=0x7f94f0141860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:27.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.707+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 shutdown_connections 2024-03-29T11:18:27.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.707+0000 7f94f6f0e640 1 -- 172.21.15.3:0/810290434 wait complete. 2024-03-29T11:18:27.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:27.707+0000 7f94f6f0e640 1 librados: shutdown 2024-03-29T11:18:27.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:34.507 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.685+0000 7fb8e9474640 10 monclient: get_monmap_and_config 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.685+0000 7fb8e9474640 10 monclient: build_initial_monmap 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.685+0000 7fb8e9474640 10 monclient: monmap: 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:34.686154+0000 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:34.686154+0000 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:34.662 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.663 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:34.663 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:34.663 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:34.663 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.687+0000 7fb8e9474640 1 Processor -- start 2024-03-29T11:18:34.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.687+0000 7fb8e9474640 1 -- start start 2024-03-29T11:18:34.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.687+0000 7fb8e9474640 10 monclient: init 2024-03-29T11:18:34.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:34.665 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e40b1f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e4146400 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e414fd10 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e406ed20 con 0x7fb8e414fd10 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e406ef90 con 0x7fb8e40b1f20 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e40b42e0 con 0x7fb8e4146400 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:34.666 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.688+0000 7fb8e9474640 10 monclient(hunting): authenticate will time out at 1041.942289s 2024-03-29T11:18:34.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.694+0000 7fb8e27fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fb8e4146400 0x7fb8e4146880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55693/0 (socket says 172.21.15.3:55693) 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.694+0000 7fb8e27fc640 1 -- 172.21.15.3:0/4225515991 learned_addr learned my addr 172.21.15.3:0/4225515991 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028581225 0 0) 0x7fb8e406ed20 con 0x7fb8e414fd10 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8cc003520 con 0x7fb8e414fd10 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392838971 0 0) 0x7fb8e40b42e0 con 0x7fb8e4146400 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:18:34.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.695+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8e406ed20 con 0x7fb8e4146400 2024-03-29T11:18:34.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 546774001 0 0) 0x7fb8cc003520 con 0x7fb8e414fd10 2024-03-29T11:18:34.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 >> v1:172.21.15.3:6790/0 conn(0x7fb8e4146400 legacy=0x7fb8e4146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 >> v1:172.21.15.32:6789/0 conn(0x7fb8e40b1f20 legacy=0x7fb8e41434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:34.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 monclient: found mon.noname-a 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e40b3690 con 0x7fb8e414fd10 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e9474640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e9474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb8d80015c0 con 0x7fb8e414fd10 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: dump: 2024-03-29T11:18:34.677 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e9474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d80017e0 con 0x7fb8e414fd10 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e9474640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:34.678 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e9474640 4 set_mon_vals no callback set 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 1 -- 172.21.15.3:0/4225515991 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb8d8001b70 con 0x7fb8e414fd10 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.696+0000 7fb8e1ffb640 10 monclient: dump: 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:34.679 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 10 monclient: shutdown 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 1 -- 172.21.15.3:0/4225515991 >> v1:172.21.15.3:6789/0 conn(0x7fb8e414fd10 legacy=0x7fb8e4152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 1 -- 172.21.15.3:0/4225515991 shutdown_connections 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 1 -- 172.21.15.3:0/4225515991 >> 172.21.15.3:0/4225515991 conn(0x7fb8e4066a00 msgr2=0x7fb8e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 1 -- 172.21.15.3:0/4225515991 shutdown_connections 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.697+0000 7fb8e9474640 1 -- 172.21.15.3:0/4225515991 wait complete. 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.698+0000 7fb8e9474640 10 monclient: build_initial_monmap 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.698+0000 7fb8e9474640 10 monclient: monmap: 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:34.680 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.698+0000 7fb8e9474640 1 librados: starting msgr at 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.698+0000 7fb8e9474640 1 librados: starting objecter 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 1 Processor -- start 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 1 -- start start 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 1 librados: setting wanted keys 2024-03-29T11:18:34.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 1 librados: calling monclient init 2024-03-29T11:18:34.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 10 monclient: init 2024-03-29T11:18:34.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.699+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-b con 0x7fb8e40b1f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-a con 0x7fb8e4146400 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 10 monclient(hunting): picked mon.noname-c con 0x7fb8e414fd10 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e414a010 con 0x7fb8e4146400 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4147c20 con 0x7fb8e40b1f20 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8e4147db0 con 0x7fb8e414fd10 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.700+0000 7fb8e9474640 10 monclient(hunting): authenticate will time out at 1041.954044s 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.704+0000 7fb8e37fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fb8e414fd10 0x7fb8e408a080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55769/0 (socket says 172.21.15.3:55769) 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.704+0000 7fb8e37fe640 1 -- 172.21.15.3:0/1091268682 learned_addr learned my addr 172.21.15.3:0/1091268682 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.705+0000 7fb8cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb8e4146400 2024-03-29T11:18:34.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.705+0000 7fb8cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:34.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579104902 0 0) 0x7fb8e414a010 con 0x7fb8e4146400 2024-03-29T11:18:34.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d0004eb0 con 0x7fb8e4146400 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3616698515 0 0) 0x7fb8d0004eb0 con 0x7fb8e4146400 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8d0005300 con 0x7fb8e4146400 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb8d4001850 con 0x7fb8e4146400 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:34.688 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.707+0000 7fb8cb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 client.?.objecter ms_handle_connect 0x7fb8e40b1f20 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2925651995 0 0) 0x7fb8d0005300 con 0x7fb8e4146400 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 >> v1:172.21.15.3:6790/0 conn(0x7fb8e414fd10 legacy=0x7fb8e408a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 >> v1:172.21.15.32:6789/0 conn(0x7fb8e40b1f20 legacy=0x7fb8e4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 1 monclient: found mon.a 2024-03-29T11:18:34.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4134650 con 0x7fb8e4146400 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 10 monclient: _renew_subs 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e4134840 con 0x7fb8e4146400 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 10 monclient: _renew_subs 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e4133e00 con 0x7fb8e4146400 2024-03-29T11:18:34.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.708+0000 7fb8e9474640 1 librados: init done 2024-03-29T11:18:34.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.709+0000 7fb8e9474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.709+0000 7fb8e9474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.709+0000 7fb8e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:18:34.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.709+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e41482f0 con 0x7fb8e4146400 2024-03-29T11:18:34.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8d40047e0 con 0x7fb8e4146400 2024-03-29T11:18:34.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:34.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb8d4001850 con 0x7fb8e4146400 2024-03-29T11:18:34.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:34.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:34.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8cb7fe640 10 monclient: dump: 2024-03-29T11:18:34.693 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:34.693 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.712+0000 7fb8e0ff9640 4 set_mon_vals no callback set 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.714+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fb8d401a800 con 0x7fb8e4146400 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.715+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fb8d4084510 con 0x7fb8e4146400 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.715+0000 7fb8cb7fe640 10 client.4362.objecter ms_dispatch 0x7fb8e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.715+0000 7fb8cb7fe640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:34.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.715+0000 7fb8cb7fe640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:34.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.715+0000 7fb8cb7fe640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:18:34.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.716+0000 7fb8e2ffd640 10 monclient: get_auth_request con 0x7fb8d006e090 auth_method 0 2024-03-29T11:18:34.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.716+0000 7fb8cb7fe640 10 client.4362.objecter ms_handle_connect 0x7fb8d006e090 2024-03-29T11:18:34.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.721+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8d404fdf0 con 0x7fb8e4146400 2024-03-29T11:18:34.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.721+0000 7fb8cb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:34.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:34.721+0000 7fb8cb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:35.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.052+0000 7fb8e9474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.052+0000 7fb8e9474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.052+0000 7fb8e9474640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:18:35.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.052+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8e406a400 con 0x7fb8e4146400 2024-03-29T11:18:35.032 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:35.032 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:35.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.055+0000 7fb8cb7fe640 1 -- 172.21.15.3:0/1091268682 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fb8d40016b0 con 0x7fb8e4146400 2024-03-29T11:18:35.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.055+0000 7fb8cb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:35.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.055+0000 7fb8cb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:35.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 10 librados: watch_flush enter 2024-03-29T11:18:35.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 10 librados: watch_flush exit 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 >> v1:172.21.15.3:6816/844689376 conn(0x7fb8d006e090 legacy=0x7fb8d0070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 10 monclient: shutdown 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 >> v1:172.21.15.3:6789/0 conn(0x7fb8e4146400 legacy=0x7fb8e41485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:35.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 shutdown_connections 2024-03-29T11:18:35.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 >> 172.21.15.3:0/1091268682 conn(0x7fb8e4066a00 msgr2=0x7fb8e4069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:35.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 shutdown_connections 2024-03-29T11:18:35.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 -- 172.21.15.3:0/1091268682 wait complete. 2024-03-29T11:18:35.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:35.059+0000 7fb8e9474640 1 librados: shutdown 2024-03-29T11:18:35.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:41.864 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:42.018 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.041+0000 7f8a097c9640 10 monclient: get_monmap_and_config 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.041+0000 7f8a097c9640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.041+0000 7f8a097c9640 10 monclient: monmap: 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:42.042721+0000 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:42.042721+0000 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:42.019 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 1 Processor -- start 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 1 -- start start 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient: init 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:42.020 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8a040b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8a040b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8a040b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040aab60 con 0x7f8a040b3560 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040aae40 con 0x7f8a040b28d0 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a040ab120 con 0x7f8a040b2cb0 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.021 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.043+0000 7f8a097c9640 10 monclient(hunting): authenticate will time out at 1049.297944s 2024-03-29T11:18:42.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.047+0000 7f8a037fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8a040b3560 0x7f8a041428a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41617/0 (socket says 172.21.15.3:41617) 2024-03-29T11:18:42.024 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.047+0000 7f8a037fe640 1 -- 172.21.15.3:0/2178064558 learned_addr learned my addr 172.21.15.3:0/2178064558 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:42.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.050+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855352055 0 0) 0x7f8a040aab60 con 0x7f8a040b3560 2024-03-29T11:18:42.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.050+0000 7f8a01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.027 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.050+0000 7f8a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.050+0000 7f8a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.050+0000 7f8a01ffb640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:18:42.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89ec003520 con 0x7f8a040b3560 2024-03-29T11:18:42.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978360940 0 0) 0x7f8a040aae40 con 0x7f8a040b28d0 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a040aab60 con 0x7f8a040b28d0 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729772069 0 0) 0x7f8a040ab120 con 0x7f8a040b2cb0 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.051+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a040aae40 con 0x7f8a040b2cb0 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203414685 0 0) 0x7f8a040aae40 con 0x7f8a040b2cb0 2024-03-29T11:18:42.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 >> v1:172.21.15.32:6789/0 conn(0x7f8a040b28d0 legacy=0x7f8a0413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 >> v1:172.21.15.3:6789/0 conn(0x7f8a040b3560 legacy=0x7f8a041428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 monclient: found mon.noname-b 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a0413c380 con 0x7f8a040b2cb0 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f89f4001530 con 0x7f8a040b2cb0 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a097c9640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.052+0000 7f8a097c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.053+0000 7f8a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.053+0000 7f8a01ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.053+0000 7f8a01ffb640 10 monclient: dump: 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:42.033 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89f4001710 con 0x7f8a040b2cb0 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a097c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a01ffb640 1 -- 172.21.15.3:0/2178064558 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f89f4001ae0 con 0x7f8a040b2cb0 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a097c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a097c9640 4 set_mon_vals no callback set 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.056+0000 7f8a01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.057+0000 7f8a01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.057+0000 7f8a01ffb640 10 monclient: dump: 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.035 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.057+0000 7f8a097c9640 10 monclient: shutdown 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.057+0000 7f8a097c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.057+0000 7f8a097c9640 1 -- 172.21.15.3:0/2178064558 >> v1:172.21.15.3:6790/0 conn(0x7f8a040b2cb0 legacy=0x7f8a04142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.058+0000 7f8a097c9640 1 -- 172.21.15.3:0/2178064558 shutdown_connections 2024-03-29T11:18:42.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.058+0000 7f8a097c9640 1 -- 172.21.15.3:0/2178064558 >> 172.21.15.3:0/2178064558 conn(0x7f8a04066a00 msgr2=0x7f8a04066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.037 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.061+0000 7f8a097c9640 1 -- 172.21.15.3:0/2178064558 shutdown_connections 2024-03-29T11:18:42.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.061+0000 7f8a097c9640 1 -- 172.21.15.3:0/2178064558 wait complete. 2024-03-29T11:18:42.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.062+0000 7f8a097c9640 10 monclient: build_initial_monmap 2024-03-29T11:18:42.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.062+0000 7f8a097c9640 10 monclient: monmap: 2024-03-29T11:18:42.040 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:42.040 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.063+0000 7f8a097c9640 1 librados: starting msgr at 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.063+0000 7f8a097c9640 1 librados: starting objecter 2024-03-29T11:18:42.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.063+0000 7f8a097c9640 1 Processor -- start 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 -- start start 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 librados: setting wanted keys 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 librados: calling monclient init 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient: init 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-a con 0x7f8a040b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-b con 0x7f8a04071e00 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient(hunting): picked mon.noname-c con 0x7f8a04070ec0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0413f1c0 con 0x7f8a040b28d0 2024-03-29T11:18:42.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0413f3f0 con 0x7f8a04071e00 2024-03-29T11:18:42.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a0406dfb0 con 0x7f8a04070ec0 2024-03-29T11:18:42.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:42.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.064+0000 7f8a097c9640 10 monclient(hunting): authenticate will time out at 1049.318908s 2024-03-29T11:18:42.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.071+0000 7f8a02ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f8a040b28d0 0x7f8a040707b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:41923/0 (socket says 172.21.15.3:41923) 2024-03-29T11:18:42.047 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.071+0000 7f8a02ffd640 1 -- 172.21.15.3:0/1712648472 learned_addr learned my addr 172.21.15.3:0/1712648472 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:42.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter ms_handle_connect 0x7f8a040b28d0 2024-03-29T11:18:42.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:42.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter ms_handle_connect 0x7f8a04070ec0 2024-03-29T11:18:42.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452612917 0 0) 0x7f8a0406dfb0 con 0x7f8a04070ec0 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f0004eb0 con 0x7f8a04070ec0 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210646479 0 0) 0x7f8a0413f1c0 con 0x7f8a040b28d0 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0406dfb0 con 0x7f8a040b28d0 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter ms_handle_connect 0x7f8a04071e00 2024-03-29T11:18:42.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.072+0000 7f89eb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966274443 0 0) 0x7f8a0413f3f0 con 0x7f8a04071e00 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a0413f1c0 con 0x7f8a04071e00 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2583166831 0 0) 0x7f8a0413f1c0 con 0x7f8a04071e00 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a0413f3f0 con 0x7f8a04071e00 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f89f4001a20 con 0x7f8a04071e00 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.052 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.053 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:42.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1936654540 0 0) 0x7f89f0004eb0 con 0x7f8a04070ec0 2024-03-29T11:18:42.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a0413f1c0 con 0x7f8a04070ec0 2024-03-29T11:18:42.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f89f8001730 con 0x7f8a04070ec0 2024-03-29T11:18:42.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 10 monclient(hunting): dump: 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1229666676 0 0) 0x7f8a0413f3f0 con 0x7f8a04071e00 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.073+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 >> v1:172.21.15.3:6790/0 conn(0x7f8a04070ec0 legacy=0x7f8a041e6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 >> v1:172.21.15.3:6789/0 conn(0x7f8a040b28d0 legacy=0x7f8a040707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 1 monclient: found mon.b 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a040718e0 con 0x7f8a04071e00 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 10 monclient: _renew_subs 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a040715b0 con 0x7f8a04071e00 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 10 monclient: _renew_subs 2024-03-29T11:18:42.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a041462c0 con 0x7f8a04071e00 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f8a097c9640 1 librados: init done 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89f4001070 con 0x7f8a04071e00 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f89f4001ba0 con 0x7f8a04071e00 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.074+0000 7f89eb7fe640 10 monclient: dump: 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:42.057 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.076+0000 7f8a017fa640 4 set_mon_vals no callback set 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.078+0000 7f8a097c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.078+0000 7f8a097c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.078+0000 7f8a097c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.078+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f89f4003a40 con 0x7f8a04071e00 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.078+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89e4005080 con 0x7f8a04071e00 2024-03-29T11:18:42.058 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f89f4083890 con 0x7f8a04071e00 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f89eb7fe640 10 client.4387.objecter ms_dispatch 0x7f8a04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f89eb7fe640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f89eb7fe640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f89eb7fe640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.079+0000 7f8a02ffd640 10 monclient: get_auth_request con 0x7f89f006e010 auth_method 0 2024-03-29T11:18:42.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.080+0000 7f89eb7fe640 10 client.4387.objecter ms_handle_connect 0x7f89f006e010 2024-03-29T11:18:42.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.088+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f89f404f170 con 0x7f8a04071e00 2024-03-29T11:18:42.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.088+0000 7f89eb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:42.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.088+0000 7f89eb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:42.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.417+0000 7f8a097c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.417+0000 7f8a097c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.417+0000 7f8a097c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:42.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.417+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89e4005330 con 0x7f8a04071e00 2024-03-29T11:18:42.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.419+0000 7f89eb7fe640 1 -- 172.21.15.3:0/1712648472 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f89f40510a0 con 0x7f8a04071e00 2024-03-29T11:18:42.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.420+0000 7f89eb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:42.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.420+0000 7f89eb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:42.397 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:42.398 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:42.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.424+0000 7f8a097c9640 10 librados: watch_flush enter 2024-03-29T11:18:42.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.424+0000 7f8a097c9640 10 librados: watch_flush exit 2024-03-29T11:18:42.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.424+0000 7f8a097c9640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:18:42.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.424+0000 7f8a097c9640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:18:42.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 >> v1:172.21.15.3:6816/844689376 conn(0x7f89f006e010 legacy=0x7f89f00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 10 monclient: shutdown 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 >> v1:172.21.15.32:6789/0 conn(0x7f8a04071e00 legacy=0x7f8a0413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 shutdown_connections 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 >> 172.21.15.3:0/1712648472 conn(0x7f8a04066a00 msgr2=0x7f8a04134ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:42.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 shutdown_connections 2024-03-29T11:18:42.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 -- 172.21.15.3:0/1712648472 wait complete. 2024-03-29T11:18:42.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:42.426+0000 7f8a097c9640 1 librados: shutdown 2024-03-29T11:18:42.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:49.228 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:49.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.398+0000 7feb812d9640 10 monclient: get_monmap_and_config 2024-03-29T11:18:49.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.398+0000 7feb812d9640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.376 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.398+0000 7feb812d9640 10 monclient: monmap: 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:49.399592+0000 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:49.399592+0000 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.399+0000 7feb812d9640 1 Processor -- start 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.400+0000 7feb812d9640 1 -- start start 2024-03-29T11:18:49.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.400+0000 7feb812d9640 10 monclient: init 2024-03-29T11:18:49.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.401+0000 7feb812d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.401+0000 7feb812d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:49.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.401+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c072320 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:49.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.401+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c0afb40 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.404+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c142810 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.404+0000 7feb7b7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7feb7c142810 0x7feb7c14d940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:59539/0 (socket says 172.21.15.3:59539) 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.404+0000 7feb7b7fe640 1 -- 172.21.15.3:0/3527300333 learned_addr learned my addr 172.21.15.3:0/3527300333 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.404+0000 7feb812d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0b01f0 con 0x7feb7c0afb40 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.405+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0aae70 con 0x7feb7c072320 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.405+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c0a9a50 con 0x7feb7c142810 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.405+0000 7feb812d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.405+0000 7feb812d9640 10 monclient(hunting): authenticate will time out at 1056.659556s 2024-03-29T11:18:49.388 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.408+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989326321 0 0) 0x7feb7c0a9a50 con 0x7feb7c142810 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb64003520 con 0x7feb7c142810 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364086276 0 0) 0x7feb7c0b01f0 con 0x7feb7c0afb40 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb7c0a9a50 con 0x7feb7c0afb40 2024-03-29T11:18:49.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 305366017 0 0) 0x7feb64003520 con 0x7feb7c142810 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 >> v1:172.21.15.32:6789/0 conn(0x7feb7c072320 legacy=0x7feb7c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 >> v1:172.21.15.3:6789/0 conn(0x7feb7c0afb40 legacy=0x7feb7c142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 monclient: found mon.noname-b 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:18:49.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c13fc90 con 0x7feb7c142810 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feb70001690 con 0x7feb7c142810 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: dump: 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:49.391 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb70001870 con 0x7feb7c142810 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb812d9640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb812d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb812d9640 4 set_mon_vals no callback set 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 1 -- 172.21.15.3:0/3527300333 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feb70001c00 con 0x7feb7c142810 2024-03-29T11:18:49.392 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.409+0000 7feb79ffb640 10 monclient: dump: 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.410+0000 7feb812d9640 10 monclient: shutdown 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.410+0000 7feb812d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:49.393 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.410+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 >> v1:172.21.15.3:6790/0 conn(0x7feb7c142810 legacy=0x7feb7c14d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.410+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 shutdown_connections 2024-03-29T11:18:49.394 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.410+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 >> 172.21.15.3:0/3527300333 conn(0x7feb7c0b3c10 msgr2=0x7feb7c0b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.413+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 shutdown_connections 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.414+0000 7feb812d9640 1 -- 172.21.15.3:0/3527300333 wait complete. 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.415+0000 7feb812d9640 10 monclient: build_initial_monmap 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.415+0000 7feb812d9640 10 monclient: monmap: 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:49.396 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.415+0000 7feb812d9640 1 librados: starting msgr at 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.415+0000 7feb812d9640 1 librados: starting objecter 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.416+0000 7feb812d9640 1 Processor -- start 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.416+0000 7feb812d9640 1 -- start start 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.416+0000 7feb812d9640 1 librados: setting wanted keys 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.416+0000 7feb812d9640 1 librados: calling monclient init 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.416+0000 7feb812d9640 10 monclient: init 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.417+0000 7feb812d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.417+0000 7feb812d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:49.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.417+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c072320 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c0afb40 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c142810 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c06e010 con 0x7feb7c072320 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c06f4e0 con 0x7feb7c142810 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb7c06f7c0 con 0x7feb7c0afb40 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.422+0000 7feb812d9640 10 monclient(hunting): authenticate will time out at 1056.676303s 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb7b7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feb7c142810 0x7feb7c1e7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:47611/0 (socket says 172.21.15.3:47611) 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb7b7fe640 1 -- 172.21.15.3:0/2756197455 learned_addr learned my addr 172.21.15.3:0/2756197455 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 client.?.objecter ms_handle_connect 0x7feb7c142810 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:49.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117995431 0 0) 0x7feb7c06f4e0 con 0x7feb7c142810 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.425+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb7c142810 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 client.?.objecter ms_handle_connect 0x7feb7c072320 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 client.?.objecter ms_handle_connect 0x7feb7c0afb40 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547170261 0 0) 0x7feb68004eb0 con 0x7feb7c142810 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb7c06f4e0 con 0x7feb7c142810 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feb70001c40 con 0x7feb7c142810 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): dump: 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336045971 0 0) 0x7feb7c06e010 con 0x7feb7c072320 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:49.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68004eb0 con 0x7feb7c072320 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2021985764 0 0) 0x7feb7c06f4e0 con 0x7feb7c142810 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.426+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 >> v1:172.21.15.3:6790/0 conn(0x7feb7c0afb40 legacy=0x7feb7c1e4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 >> v1:172.21.15.3:6789/0 conn(0x7feb7c072320 legacy=0x7feb7c144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 1 monclient: found mon.b 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb7c06dca0 con 0x7feb7c142810 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:49.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 10 monclient: _renew_subs 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb7c1e4d60 con 0x7feb7c142810 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 10 monclient: _renew_subs 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb7c06e630 con 0x7feb7c142810 2024-03-29T11:18:49.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.427+0000 7feb812d9640 1 librados: init done 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb70004a90 con 0x7feb7c142810 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feb70004ea0 con 0x7feb7c142810 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.430+0000 7feb637fe640 10 monclient: dump: 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.415 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.431+0000 7feb797fa640 4 set_mon_vals no callback set 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.431+0000 7feb812d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.431+0000 7feb812d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.432+0000 7feb812d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.432+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c005080 con 0x7feb7c142810 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.432+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7feb700050c0 con 0x7feb7c142810 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7feb70084be0 con 0x7feb7c142810 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb637fe640 10 client.4393.objecter ms_dispatch 0x7feb7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:49.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb637fe640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:49.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb637fe640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:49.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb637fe640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:18:49.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.433+0000 7feb7affd640 10 monclient: get_auth_request con 0x7feb6806db50 auth_method 0 2024-03-29T11:18:49.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.434+0000 7feb637fe640 10 client.4393.objecter ms_handle_connect 0x7feb6806db50 2024-03-29T11:18:49.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.442+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb700504c0 con 0x7feb7c142810 2024-03-29T11:18:49.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.442+0000 7feb637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:49.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.442+0000 7feb637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.780+0000 7feb812d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.780+0000 7feb812d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.780+0000 7feb812d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:18:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.780+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb5c005330 con 0x7feb7c142810 2024-03-29T11:18:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.781+0000 7feb637fe640 1 -- 172.21.15.3:0/2756197455 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7feb700523f0 con 0x7feb7c142810 2024-03-29T11:18:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.781+0000 7feb637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.781+0000 7feb637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:49.758 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:49.758 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:49.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.784+0000 7feb812d9640 10 librados: watch_flush enter 2024-03-29T11:18:49.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 10 librados: watch_flush exit 2024-03-29T11:18:49.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 >> v1:172.21.15.3:6816/844689376 conn(0x7feb6806db50 legacy=0x7feb68070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 10 monclient: shutdown 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 >> v1:172.21.15.32:6789/0 conn(0x7feb7c142810 legacy=0x7feb7c1e7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 shutdown_connections 2024-03-29T11:18:49.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.785+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 >> 172.21.15.3:0/2756197455 conn(0x7feb7c0b3c10 msgr2=0x7feb7c13f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:49.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.786+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 shutdown_connections 2024-03-29T11:18:49.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.786+0000 7feb812d9640 1 -- 172.21.15.3:0/2756197455 wait complete. 2024-03-29T11:18:49.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:49.786+0000 7feb812d9640 1 librados: shutdown 2024-03-29T11:18:49.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:56.590 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:56.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.763+0000 7f32251c6640 10 monclient: get_monmap_and_config 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.763+0000 7f32251c6640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.763+0000 7f32251c6640 10 monclient: monmap: 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:18:56.764048+0000 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:18:56.764048+0000 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:18:56.740 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:18:56.741 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.764+0000 7f32251c6640 1 Processor -- start 2024-03-29T11:18:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.764+0000 7f32251c6640 1 -- start start 2024-03-29T11:18:56.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.764+0000 7f32251c6640 10 monclient: init 2024-03-29T11:18:56.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:56.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-c con 0x7f32200b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-b con 0x7f32200b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-a con 0x7f32200b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200aab60 con 0x7f32200b3560 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200aae40 con 0x7f32200b28d0 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32200ab120 con 0x7f32200b2cb0 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.765+0000 7f32251c6640 10 monclient(hunting): authenticate will time out at 1064.019883s 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.766+0000 7f321ed9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f32200b28d0 0x7f322013bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:48166/0 (socket says 172.21.15.3:48166) 2024-03-29T11:18:56.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.766+0000 7f321ed9d640 1 -- 172.21.15.3:0/4037201372 learned_addr learned my addr 172.21.15.3:0/4037201372 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:56.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321100118 0 0) 0x7f32200aab60 con 0x7f32200b3560 2024-03-29T11:18:56.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208003520 con 0x7f32200b3560 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598631499 0 0) 0x7f32200aae40 con 0x7f32200b28d0 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32200aab60 con 0x7f32200b28d0 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373563269 0 0) 0x7f3208003520 con 0x7f32200b3560 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 >> v1:172.21.15.3:6790/0 conn(0x7f32200b2cb0 legacy=0x7f3220142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 >> v1:172.21.15.32:6789/0 conn(0x7f32200b28d0 legacy=0x7f322013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 monclient: found mon.noname-a 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322013c380 con 0x7f32200b3560 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f32251c6640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f32251c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3214001690 con 0x7f32200b3560 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f321dd9b640 10 monclient: dump: 2024-03-29T11:18:56.746 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.767+0000 7f32251c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:56.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32140018b0 con 0x7f32200b3560 2024-03-29T11:18:56.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 1 -- 172.21.15.3:0/4037201372 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3214001c40 con 0x7f32200b3560 2024-03-29T11:18:56.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f32251c6640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f32251c6640 4 set_mon_vals no callback set 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.768+0000 7f321dd9b640 10 monclient: dump: 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:56.749 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 10 monclient: shutdown 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 1 -- 172.21.15.3:0/4037201372 >> v1:172.21.15.3:6789/0 conn(0x7f32200b3560 legacy=0x7f32201428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 1 -- 172.21.15.3:0/4037201372 shutdown_connections 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 1 -- 172.21.15.3:0/4037201372 >> 172.21.15.3:0/4037201372 conn(0x7f3220066a00 msgr2=0x7f3220066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 1 -- 172.21.15.3:0/4037201372 shutdown_connections 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.770+0000 7f32251c6640 1 -- 172.21.15.3:0/4037201372 wait complete. 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.771+0000 7f32251c6640 10 monclient: build_initial_monmap 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.771+0000 7f32251c6640 10 monclient: monmap: 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:18:56.750 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:56.751 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.751 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:18:56.751 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:18:56.751 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:18:56.751 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 librados: starting msgr at 2024-03-29T11:18:56.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 librados: starting objecter 2024-03-29T11:18:56.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 Processor -- start 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 -- start start 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 librados: setting wanted keys 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 1 librados: calling monclient init 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.772+0000 7f32251c6640 10 monclient: init 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.773+0000 7f32251c6640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-c con 0x7f32200b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3220145290 addr v1:172.21.15.32:6789/0 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient(hunting): picked mon.noname-a con 0x7f322013ec70 addr v1:172.21.15.3:6789/0 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322013e7e0 con 0x7f322013ec70 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322006e270 con 0x7f3220145290 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322013f0e0 con 0x7f32200b28d0 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:56.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32251c6640 10 monclient(hunting): authenticate will time out at 1064.028358s 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f321e59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3220145290 0x7f3220131350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:49932/0 (socket says 172.21.15.3:49932) 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f321e59c640 1 -- 172.21.15.3:0/835427683 learned_addr learned my addr 172.21.15.3:0/835427683 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32077fe640 10 client.?.objecter ms_handle_connect 0x7f3220145290 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.774+0000 7f32077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 client.?.objecter ms_handle_connect 0x7f32200b28d0 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244417030 0 0) 0x7f322006e270 con 0x7f3220145290 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f320c004eb0 con 0x7f3220145290 2024-03-29T11:18:56.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178650871 0 0) 0x7f322013f0e0 con 0x7f32200b28d0 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f322006e270 con 0x7f32200b28d0 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 client.?.objecter ms_handle_connect 0x7f322013ec70 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.775+0000 7f32077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 554209985 0 0) 0x7f322006e270 con 0x7f32200b28d0 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f322013f0e0 con 0x7f32200b28d0 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3210001930 con 0x7f32200b28d0 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): dump: 2024-03-29T11:18:56.760 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.761 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830959881 0 0) 0x7f320c004eb0 con 0x7f3220145290 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f322006e270 con 0x7f3220145290 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3218001db0 con 0x7f3220145290 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.776+0000 7f32077fe640 10 monclient(hunting): dump: 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.762 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1692794090 0 0) 0x7f322013f0e0 con 0x7f32200b28d0 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 >> v1:172.21.15.32:6789/0 conn(0x7f3220145290 legacy=0x7f3220131350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 >> v1:172.21.15.3:6789/0 conn(0x7f322013ec70 legacy=0x7f3220131a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 1 monclient: found mon.c 2024-03-29T11:18:56.763 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32201df650 con 0x7f32200b28d0 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 10 monclient: _finish_auth 0 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 10 monclient: _renew_subs 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:56.764 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3220132170 con 0x7f32200b28d0 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 10 monclient: _renew_subs 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f322013dac0 con 0x7f32200b28d0 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.777+0000 7f32251c6640 1 librados: init done 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3210001d40 con 0x7f32200b28d0 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3210004a40 con 0x7f32200b28d0 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:56.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.778+0000 7f32077fe640 10 monclient: dump: 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:18:56.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.779+0000 7f321cd99640 4 set_mon_vals no callback set 2024-03-29T11:18:56.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.781+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f3210004ba0 con 0x7f32200b28d0 2024-03-29T11:18:56.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.781+0000 7f32057fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.781+0000 7f32057fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.781+0000 7f32057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:56.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.781+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322006a400 con 0x7f32200b28d0 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f32100849e0 con 0x7f32200b28d0 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f32077fe640 10 client.4397.objecter ms_dispatch 0x7f3220000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f32077fe640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f32077fe640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f321e59c640 10 monclient: get_auth_request con 0x7f320c06de30 auth_method 0 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.782+0000 7f32077fe640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.783+0000 7f32077fe640 10 client.4397.objecter ms_handle_connect 0x7f320c06de30 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.791+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32100502c0 con 0x7f32200b28d0 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.791+0000 7f32077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:56.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:56.791+0000 7f32077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.143+0000 7f32057fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.143+0000 7f32057fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.143+0000 7f32057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:18:57.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.143+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f322013e5b0 con 0x7f32200b28d0 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.144+0000 7f32077fe640 1 -- 172.21.15.3:0/835427683 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f32100521f0 con 0x7f32200b28d0 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.144+0000 7f32077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.144+0000 7f32077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:18:57.121 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 10 librados: watch_flush enter 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 10 librados: watch_flush exit 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 10 client.4397.objecter successfully canceled tick 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 >> v1:172.21.15.3:6816/844689376 conn(0x7f320c06de30 legacy=0x7f320c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 10 monclient: shutdown 2024-03-29T11:18:57.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.147+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 >> v1:172.21.15.3:6790/0 conn(0x7f32200b28d0 legacy=0x7f32201def40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.148+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 shutdown_connections 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.148+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 >> 172.21.15.3:0/835427683 conn(0x7f3220066a00 msgr2=0x7f3220135080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.148+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 shutdown_connections 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.148+0000 7f32057fa640 1 -- 172.21.15.3:0/835427683 wait complete. 2024-03-29T11:18:57.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:18:57.148+0000 7f32057fa640 1 librados: shutdown 2024-03-29T11:18:57.142 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:03.949 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:04.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.137+0000 7f86e277b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:04.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.137+0000 7f86e277b640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.137+0000 7f86e277b640 10 monclient: monmap: 2024-03-29T11:19:04.114 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:04.114 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:04.138585+0000 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:04.138585+0000 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.139+0000 7f86e277b640 1 Processor -- start 2024-03-29T11:19:04.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.139+0000 7f86e277b640 1 -- start start 2024-03-29T11:19:04.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.139+0000 7f86e277b640 10 monclient: init 2024-03-29T11:19:04.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc0b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc0b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc0b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0aab60 con 0x7f86dc0b3560 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0aae40 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc0ab120 con 0x7f86dc0b28d0 2024-03-29T11:19:04.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86e277b640 10 monclient(hunting): authenticate will time out at 1071.395544s 2024-03-29T11:19:04.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86dbfff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f86dc0b28d0 0x7f86dc13bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42388/0 (socket says 172.21.15.3:42388) 2024-03-29T11:19:04.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.141+0000 7f86dbfff640 1 -- 172.21.15.3:0/4018189346 learned_addr learned my addr 172.21.15.3:0/4018189346 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:04.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243900700 0 0) 0x7f86dc0aae40 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8003520 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.142+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732089227 0 0) 0x7f86dc0ab120 con 0x7f86dc0b28d0 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc0aae40 con 0x7f86dc0b28d0 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489713265 0 0) 0x7f86dc0aab60 con 0x7f86dc0b3560 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.120 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:19:04.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc0ab120 con 0x7f86dc0b3560 2024-03-29T11:19:04.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008008945 0 0) 0x7f86c8003520 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 >> v1:172.21.15.3:6790/0 conn(0x7f86dc0b28d0 legacy=0x7f86dc13bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 >> v1:172.21.15.3:6789/0 conn(0x7f86dc0b3560 legacy=0x7f86dc1428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 monclient: found mon.noname-c 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc13c380 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.143+0000 7f86daffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86cc001690 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86e277b640 5 monclient: authenticate success, global_id 4402 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86e277b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:04.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: dump: 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86e277b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86cc0018b0 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 1 -- 172.21.15.3:0/4018189346 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86cc001c40 con 0x7f86dc0b2cb0 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86daffd640 10 monclient: dump: 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86e277b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.144+0000 7f86e277b640 4 set_mon_vals no callback set 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.145+0000 7f86e277b640 10 monclient: shutdown 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.145+0000 7f86e277b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.145+0000 7f86e277b640 1 -- 172.21.15.3:0/4018189346 >> v1:172.21.15.32:6789/0 conn(0x7f86dc0b2cb0 legacy=0x7f86dc142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.145+0000 7f86e277b640 1 -- 172.21.15.3:0/4018189346 shutdown_connections 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.145+0000 7f86e277b640 1 -- 172.21.15.3:0/4018189346 >> 172.21.15.3:0/4018189346 conn(0x7f86dc066a00 msgr2=0x7f86dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.146+0000 7f86e277b640 1 -- 172.21.15.3:0/4018189346 shutdown_connections 2024-03-29T11:19:04.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.146+0000 7f86e277b640 1 -- 172.21.15.3:0/4018189346 wait complete. 2024-03-29T11:19:04.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.147+0000 7f86e277b640 10 monclient: build_initial_monmap 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.147+0000 7f86e277b640 10 monclient: monmap: 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 librados: starting msgr at 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 librados: starting objecter 2024-03-29T11:19:04.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 Processor -- start 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 -- start start 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 librados: setting wanted keys 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 1 librados: calling monclient init 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.148+0000 7f86e277b640 10 monclient: init 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-a con 0x7f86dc0b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-c con 0x7f86dc0b88e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient(hunting): picked mon.noname-b con 0x7f86dc145fa0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc13da70 con 0x7f86dc0b28d0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc146380 con 0x7f86dc145fa0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc13f830 con 0x7f86dc0b88e0 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:04.130 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86e277b640 10 monclient(hunting): authenticate will time out at 1071.404577s 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86db7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f86dc0b88e0 0x7f86dc1ded40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:42678/0 (socket says 172.21.15.3:42678) 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.150+0000 7f86db7fe640 1 -- 172.21.15.3:0/768189234 learned_addr learned my addr 172.21.15.3:0/768189234 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter ms_handle_connect 0x7f86dc0b28d0 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter ms_handle_connect 0x7f86dc0b88e0 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter ms_handle_connect 0x7f86dc145fa0 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720836171 0 0) 0x7f86dc13f830 con 0x7f86dc0b88e0 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86bc004eb0 con 0x7f86dc0b88e0 2024-03-29T11:19:04.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422566482 0 0) 0x7f86dc13da70 con 0x7f86dc0b28d0 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc13f830 con 0x7f86dc0b28d0 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053216387 0 0) 0x7f86dc146380 con 0x7f86dc145fa0 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.151+0000 7f86d8ff9640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:19:04.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.152+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86dc13da70 con 0x7f86dc145fa0 2024-03-29T11:19:04.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.152+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 897511253 0 0) 0x7f86bc004eb0 con 0x7f86dc0b88e0 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86dc146380 con 0x7f86dc0b88e0 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1807104072 0 0) 0x7f86dc13f830 con 0x7f86dc0b28d0 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86bc004eb0 con 0x7f86dc0b28d0 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86cc0015d0 con 0x7f86dc0b88e0 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.136 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369957377 0 0) 0x7f86dc13da70 con 0x7f86dc145fa0 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86dc13f830 con 0x7f86dc145fa0 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86d4001c70 con 0x7f86dc0b28d0 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.137 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86c40017d0 con 0x7f86dc145fa0 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.138 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4147013356 0 0) 0x7f86bc004eb0 con 0x7f86dc0b28d0 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 >> v1:172.21.15.3:6790/0 conn(0x7f86dc0b88e0 legacy=0x7f86dc1ded40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 >> v1:172.21.15.32:6789/0 conn(0x7f86dc145fa0 legacy=0x7f86dc1e2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.153+0000 7f86d8ff9640 1 monclient: found mon.a 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc13c9a0 con 0x7f86dc0b28d0 2024-03-29T11:19:04.139 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 5 monclient: authenticate success, global_id 4386 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 10 monclient: _renew_subs 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc13e5a0 con 0x7f86dc0b28d0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 10 monclient: _renew_subs 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc13c630 con 0x7f86dc0b28d0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.154+0000 7f86e277b640 1 librados: init done 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86d4004b60 con 0x7f86dc0b28d0 2024-03-29T11:19:04.140 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f86d4004e70 con 0x7f86dc0b28d0 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d8ff9640 10 monclient: dump: 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:04.141 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.155+0000 7f86d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:19:04.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.156+0000 7f86e277b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.156+0000 7f86e277b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.156+0000 7f86e277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:04.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.156+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c0005080 con 0x7f86dc0b28d0 2024-03-29T11:19:04.142 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.158+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f86d401ae40 con 0x7f86dc0b28d0 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.159+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f86d4084c40 con 0x7f86dc0b28d0 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.159+0000 7f86d8ff9640 10 client.4386.objecter ms_dispatch 0x7f86dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.159+0000 7f86d8ff9640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.159+0000 7f86d8ff9640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.159+0000 7f86d8ff9640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.160+0000 7f86db7fe640 10 monclient: get_auth_request con 0x7f86bc06e0b0 auth_method 0 2024-03-29T11:19:04.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.160+0000 7f86d8ff9640 10 client.4386.objecter ms_handle_connect 0x7f86bc06e0b0 2024-03-29T11:19:04.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.168+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86d4050520 con 0x7f86dc0b28d0 2024-03-29T11:19:04.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.168+0000 7f86d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:04.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.168+0000 7f86d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:04.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.545+0000 7f86e277b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.545+0000 7f86e277b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.545+0000 7f86e277b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:04.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.545+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c0005330 con 0x7f86dc0b28d0 2024-03-29T11:19:04.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.547+0000 7f86d8ff9640 1 -- 172.21.15.3:0/768189234 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f86d4052450 con 0x7f86dc0b28d0 2024-03-29T11:19:04.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.547+0000 7f86d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:04.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.547+0000 7f86d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:04.524 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:04.524 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:04.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 10 librados: watch_flush enter 2024-03-29T11:19:04.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 10 librados: watch_flush exit 2024-03-29T11:19:04.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T11:19:04.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 10 client.4386.objecter successfully canceled tick 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 >> v1:172.21.15.3:6816/844689376 conn(0x7f86bc06e0b0 legacy=0x7f86bc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 10 monclient: shutdown 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.551+0000 7f86e277b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 >> v1:172.21.15.3:6789/0 conn(0x7f86dc0b28d0 legacy=0x7f86dc13de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 shutdown_connections 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 >> 172.21.15.3:0/768189234 conn(0x7f86dc066a00 msgr2=0x7f86dc134e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:04.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 shutdown_connections 2024-03-29T11:19:04.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 -- 172.21.15.3:0/768189234 wait complete. 2024-03-29T11:19:04.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:04.552+0000 7f86e277b640 1 librados: shutdown 2024-03-29T11:19:04.543 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:11.350 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:11.482 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.505+0000 7f8df5cf4640 10 monclient: get_monmap_and_config 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.505+0000 7f8df5cf4640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.505+0000 7f8df5cf4640 10 monclient: monmap: 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:11.507133+0000 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:11.507133+0000 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:11.483 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.506+0000 7f8df5cf4640 1 Processor -- start 2024-03-29T11:19:11.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.506+0000 7f8df5cf4640 1 -- start start 2024-03-29T11:19:11.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.506+0000 7f8df5cf4640 10 monclient: init 2024-03-29T11:19:11.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.507+0000 7f8df5cf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.484 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.507+0000 7f8df5cf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:11.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f8df00b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:11.487 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f8df00b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f8df00b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00aab60 con 0x7f8df00b28d0 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00aae40 con 0x7f8df00b2cb0 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00ab120 con 0x7f8df00b3560 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8df5cf4640 10 monclient(hunting): authenticate will time out at 1078.764662s 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8deffff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8df00b3560 0x7f8df01428a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50456/0 (socket says 172.21.15.3:50456) 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.509+0000 7f8deffff640 1 -- 172.21.15.3:0/539227873 learned_addr learned my addr 172.21.15.3:0/539227873 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2099562007 0 0) 0x7f8df00ab120 con 0x7f8df00b3560 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.488 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:19:11.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd8003520 con 0x7f8df00b3560 2024-03-29T11:19:11.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3321185271 0 0) 0x7f8dd8003520 con 0x7f8df00b3560 2024-03-29T11:19:11.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 >> v1:172.21.15.32:6789/0 conn(0x7f8df00b2cb0 legacy=0x7f8df0142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.489 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 >> v1:172.21.15.3:6789/0 conn(0x7f8df00b28d0 legacy=0x7f8df013bc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:11.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.490 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df013c380 con 0x7f8df00b3560 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8de8001a10 con 0x7f8df00b3560 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.510+0000 7f8dee7fc640 10 monclient: dump: 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.491 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8df5cf4640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8df5cf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de8001be0 con 0x7f8df00b3560 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8df5cf4640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8df5cf4640 4 set_mon_vals no callback set 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 1 -- 172.21.15.3:0/539227873 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8de8002030 con 0x7f8df00b3560 2024-03-29T11:19:11.492 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.511+0000 7f8dee7fc640 10 monclient: dump: 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.493 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.512+0000 7f8df5cf4640 10 monclient: shutdown 2024-03-29T11:19:11.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.512+0000 7f8df5cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:11.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.512+0000 7f8df5cf4640 1 -- 172.21.15.3:0/539227873 >> v1:172.21.15.3:6790/0 conn(0x7f8df00b3560 legacy=0x7f8df01428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.513+0000 7f8df5cf4640 1 -- 172.21.15.3:0/539227873 shutdown_connections 2024-03-29T11:19:11.494 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.513+0000 7f8df5cf4640 1 -- 172.21.15.3:0/539227873 >> 172.21.15.3:0/539227873 conn(0x7f8df0066a00 msgr2=0x7f8df0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.514+0000 7f8df5cf4640 1 -- 172.21.15.3:0/539227873 shutdown_connections 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.514+0000 7f8df5cf4640 1 -- 172.21.15.3:0/539227873 wait complete. 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.515+0000 7f8df5cf4640 10 monclient: build_initial_monmap 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.515+0000 7f8df5cf4640 10 monclient: monmap: 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:11.501 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.515+0000 7f8df5cf4640 1 librados: starting msgr at 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.515+0000 7f8df5cf4640 1 librados: starting objecter 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.515+0000 7f8df5cf4640 1 Processor -- start 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.516+0000 7f8df5cf4640 1 -- start start 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.516+0000 7f8df5cf4640 1 librados: setting wanted keys 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.516+0000 7f8df5cf4640 1 librados: calling monclient init 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.516+0000 7f8df5cf4640 10 monclient: init 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:11.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-a con 0x7f8df00b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-b con 0x7f8df0136ba0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient(hunting): picked mon.noname-c con 0x7f8df01e0170 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8deffff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8df01e0170 0x7f8df01e2560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52586/0 (socket says 172.21.15.3:52586) 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8deffff640 1 -- 172.21.15.3:0/4158990452 learned_addr learned my addr 172.21.15.3:0/4158990452 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df0145410 con 0x7f8df00b28d0 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df0146140 con 0x7f8df0136ba0 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df01364a0 con 0x7f8df01e0170 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.517+0000 7f8df5cf4640 10 monclient(hunting): authenticate will time out at 1078.772980s 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.518+0000 7f8dd7fff640 10 client.?.objecter ms_handle_connect 0x7f8df01e0170 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.518+0000 7f8dd7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:11.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134483649 0 0) 0x7f8df01364a0 con 0x7f8df01e0170 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc004eb0 con 0x7f8df01e0170 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 461897337 0 0) 0x7f8ddc004eb0 con 0x7f8df01e0170 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8df01364a0 con 0x7f8df01e0170 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8de8001ef0 con 0x7f8df01e0170 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): dump: 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.504 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.519+0000 7f8dd7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 client.?.objecter ms_handle_connect 0x7f8df00b28d0 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3760802158 0 0) 0x7f8df01364a0 con 0x7f8df01e0170 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 >> v1:172.21.15.32:6789/0 conn(0x7f8df0136ba0 legacy=0x7f8df0144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 >> v1:172.21.15.3:6789/0 conn(0x7f8df00b28d0 legacy=0x7f8df0135800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 monclient: found mon.c 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:11.505 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df0136780 con 0x7f8df01e0170 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 10 monclient: _renew_subs 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df0145690 con 0x7f8df01e0170 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 10 monclient: _renew_subs 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df0144380 con 0x7f8df01e0170 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8df5cf4640 1 librados: init done 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de8004e10 con 0x7f8df01e0170 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dd7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:11.506 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.520+0000 7f8dedffb640 4 set_mon_vals no callback set 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.521+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8de8005120 con 0x7f8df01e0170 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.521+0000 7f8dd7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.521+0000 7f8dd7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.521+0000 7f8dd7fff640 10 monclient: dump: 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:11.507 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:11.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.522+0000 7f8df5cf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.522+0000 7f8df5cf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.522+0000 7f8df5cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.508 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.522+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dd0005080 con 0x7f8df01e0170 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f8de801b0f0 con 0x7f8df01e0170 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f8de8084e00 con 0x7f8df01e0170 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 10 client.4409.objecter ms_dispatch 0x7f8df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8def7fe640 10 monclient: get_auth_request con 0x7f8ddc06db80 auth_method 0 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.524+0000 7f8dd7fff640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.525+0000 7f8dd7fff640 10 client.4409.objecter ms_handle_connect 0x7f8ddc06db80 2024-03-29T11:19:11.509 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.529+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8de80506e0 con 0x7f8df01e0170 2024-03-29T11:19:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.529+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:11.510 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.529+0000 7f8dd7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.842+0000 7f8df5cf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.842+0000 7f8df5cf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.842+0000 7f8df5cf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.842+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dd0005330 con 0x7f8df01e0170 2024-03-29T11:19:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.843+0000 7f8dd7fff640 1 -- 172.21.15.3:0/4158990452 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f8de8052610 con 0x7f8df01e0170 2024-03-29T11:19:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.843+0000 7f8dd7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.843+0000 7f8dd7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:11.821 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:11.821 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 10 librados: watch_flush enter 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 10 librados: watch_flush exit 2024-03-29T11:19:11.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 >> v1:172.21.15.3:6816/844689376 conn(0x7f8ddc06db80 legacy=0x7f8ddc070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 10 monclient: shutdown 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 >> v1:172.21.15.3:6790/0 conn(0x7f8df01e0170 legacy=0x7f8df01e2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 shutdown_connections 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 >> 172.21.15.3:0/4158990452 conn(0x7f8df0066a00 msgr2=0x7f8df013b5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 shutdown_connections 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.846+0000 7f8df5cf4640 1 -- 172.21.15.3:0/4158990452 wait complete. 2024-03-29T11:19:11.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:11.847+0000 7f8df5cf4640 1 librados: shutdown 2024-03-29T11:19:11.838 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:18.646 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:18.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.815+0000 7fb59416b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:18.792 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.815+0000 7fb59416b640 10 monclient: build_initial_monmap 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.815+0000 7fb59416b640 10 monclient: monmap: 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:18.816808+0000 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:18.816808+0000 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:18.793 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.817+0000 7fb59416b640 1 Processor -- start 2024-03-29T11:19:18.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.817+0000 7fb59416b640 1 -- start start 2024-03-29T11:19:18.794 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.817+0000 7fb59416b640 10 monclient: init 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-b con 0x7fb58c0b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-c con 0x7fb58c0b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-a con 0x7fb58c0b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c0aab60 con 0x7fb58c0b3560 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c0aae40 con 0x7fb58c0b2cb0 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c0ab120 con 0x7fb58c0b28d0 2024-03-29T11:19:18.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:18.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.818+0000 7fb59416b640 10 monclient(hunting): authenticate will time out at 1086.072509s 2024-03-29T11:19:18.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.823+0000 7fb591f07640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fb58c0b28d0 0x7fb58c13bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55273/0 (socket says 172.21.15.3:55273) 2024-03-29T11:19:18.800 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.823+0000 7fb591f07640 1 -- 172.21.15.3:0/38801720 learned_addr learned my addr 172.21.15.3:0/38801720 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562266126 0 0) 0x7fb58c0ab120 con 0x7fb58c0b28d0 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 10 monclient(hunting): _negotiate 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:19:18.801 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.824+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578003520 con 0x7fb58c0b28d0 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702399604 0 0) 0x7fb578003520 con 0x7fb58c0b28d0 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 >> v1:172.21.15.32:6789/0 conn(0x7fb58c0b2cb0 legacy=0x7fb58c142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 >> v1:172.21.15.3:6789/0 conn(0x7fb58c0b3560 legacy=0x7fb58c1428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: _finish_hunting 0 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 monclient: found mon.noname-b 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:19:18.804 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c13c380 con 0x7fb58c0b28d0 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: _finish_auth 0 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: _check_auth_tickets 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb588001670 con 0x7fb58c0b28d0 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: dump: 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.805 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb59416b640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb59416b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 1 -- 172.21.15.3:0/38801720 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5880017d0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb590f05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb59416b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.827+0000 7fb59416b640 4 set_mon_vals no callback set 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.828+0000 7fb59416b640 10 monclient: shutdown 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.828+0000 7fb59416b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.828+0000 7fb59416b640 1 -- 172.21.15.3:0/38801720 >> v1:172.21.15.3:6790/0 conn(0x7fb58c0b28d0 legacy=0x7fb58c13bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.829+0000 7fb59416b640 1 -- 172.21.15.3:0/38801720 shutdown_connections 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.829+0000 7fb59416b640 1 -- 172.21.15.3:0/38801720 >> 172.21.15.3:0/38801720 conn(0x7fb58c066a00 msgr2=0x7fb58c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.830+0000 7fb59416b640 1 -- 172.21.15.3:0/38801720 shutdown_connections 2024-03-29T11:19:18.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.831+0000 7fb59416b640 1 -- 172.21.15.3:0/38801720 wait complete. 2024-03-29T11:19:18.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 10 monclient: build_initial_monmap 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 10 monclient: monmap: 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 librados: starting msgr at 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 librados: starting objecter 2024-03-29T11:19:18.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 Processor -- start 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 -- start start 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 librados: setting wanted keys 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 1 librados: calling monclient init 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.832+0000 7fb59416b640 10 monclient: init 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-c con 0x7fb58c0b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-b con 0x7fb58c06de80 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient(hunting): picked mon.noname-a con 0x7fb58c13e6b0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c13bf30 con 0x7fb58c13e6b0 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c06ec90 con 0x7fb58c06de80 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58c1329a0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:18.812 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.833+0000 7fb59416b640 10 monclient(hunting): authenticate will time out at 1086.087613s 2024-03-29T11:19:18.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.838+0000 7fb591f07640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fb58c0b28d0 0x7fb58c145ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55365/0 (socket says 172.21.15.3:55365) 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.838+0000 7fb591f07640 1 -- 172.21.15.3:0/1307150821 learned_addr learned my addr 172.21.15.3:0/1307150821 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 client.?.objecter ms_handle_connect 0x7fb58c0b28d0 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 client.?.objecter ms_handle_connect 0x7fb58c06de80 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680966662 0 0) 0x7fb58c1329a0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb570004eb0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525758787 0 0) 0x7fb58c06ec90 con 0x7fb58c06de80 2024-03-29T11:19:18.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:18.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:18.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:18.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:19:18.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c1329a0 con 0x7fb58c06de80 2024-03-29T11:19:18.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3682461163 0 0) 0x7fb58c1329a0 con 0x7fb58c06de80 2024-03-29T11:19:18.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58c06ec90 con 0x7fb58c06de80 2024-03-29T11:19:18.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346832657 0 0) 0x7fb570004eb0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb58c1329a0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb57c0018b0 con 0x7fb58c06de80 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): dump: 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:18.823 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.842+0000 7fb5827fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb5880049c0 con 0x7fb58c0b28d0 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 10 monclient(hunting): dump: 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:18.824 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2649075119 0 0) 0x7fb58c06ec90 con 0x7fb58c06de80 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 >> v1:172.21.15.3:6790/0 conn(0x7fb58c0b28d0 legacy=0x7fb58c145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.843+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 >> v1:172.21.15.3:6789/0 conn(0x7fb58c13e6b0 legacy=0x7fb58c1eed30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 1 monclient: found mon.b 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c06ef00 con 0x7fb58c06de80 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:18.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb5827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:18.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:19:18.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 10 monclient: _renew_subs 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58c06f1e0 con 0x7fb58c06de80 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 10 monclient: _renew_subs 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58c1f0450 con 0x7fb58c06de80 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.845+0000 7fb59416b640 1 librados: init done 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb577fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb577fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb57c003610 con 0x7fb58c06de80 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb577fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c005080 con 0x7fb58c06de80 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:18.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb57c003920 con 0x7fb58c06de80 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb5827fc640 10 monclient: dump: 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:18.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.847+0000 7fb583fff640 4 set_mon_vals no callback set 2024-03-29T11:19:18.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.851+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fb57c0198f0 con 0x7fb58c06de80 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb591f07640 10 monclient: get_auth_request con 0x7fb57006de50 auth_method 0 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fb57c083810 con 0x7fb58c06de80 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb5827fc640 10 client.4411.objecter ms_dispatch 0x7fb58c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb5827fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb5827fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.852+0000 7fb5827fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:19:18.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.853+0000 7fb5827fc640 10 client.4411.objecter ms_handle_connect 0x7fb57006de50 2024-03-29T11:19:18.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.856+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb57c04f0f0 con 0x7fb58c06de80 2024-03-29T11:19:18.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.856+0000 7fb5827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:18.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:18.856+0000 7fb5827fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.198+0000 7fb577fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.198+0000 7fb577fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.198+0000 7fb577fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:19.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.198+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb56c005330 con 0x7fb58c06de80 2024-03-29T11:19:19.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.201+0000 7fb5827fc640 1 -- 172.21.15.3:0/1307150821 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fb57c051020 con 0x7fb58c06de80 2024-03-29T11:19:19.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.201+0000 7fb5827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:19.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.201+0000 7fb5827fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:19.179 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:19.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.206+0000 7fb577fff640 10 librados: watch_flush enter 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 10 librados: watch_flush exit 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 >> v1:172.21.15.3:6816/844689376 conn(0x7fb57006de50 legacy=0x7fb570070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 10 monclient: shutdown 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:19.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.207+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 >> v1:172.21.15.32:6789/0 conn(0x7fb58c06de80 legacy=0x7fb58c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.208+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 shutdown_connections 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.208+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 >> 172.21.15.3:0/1307150821 conn(0x7fb58c066a00 msgr2=0x7fb58c134fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.208+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 shutdown_connections 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.208+0000 7fb577fff640 1 -- 172.21.15.3:0/1307150821 wait complete. 2024-03-29T11:19:19.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:19.208+0000 7fb577fff640 1 librados: shutdown 2024-03-29T11:19:19.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:26.009 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:26.156 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.180+0000 7f0f946fb640 10 monclient: get_monmap_and_config 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.180+0000 7f0f946fb640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.180+0000 7f0f946fb640 10 monclient: monmap: 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:26.180938+0000 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:26.180938+0000 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:26.157 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.180+0000 7f0f946fb640 1 Processor -- start 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.181+0000 7f0f946fb640 1 -- start start 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.181+0000 7f0f946fb640 10 monclient: init 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0f8c0b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:26.159 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-c con 0x7f0f8c0b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8c0b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c0aab60 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c0aae40 con 0x7f0f8c0b2cb0 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c0ab120 con 0x7f0f8c0b3560 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.160 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.182+0000 7f0f946fb640 10 monclient(hunting): authenticate will time out at 1093.436432s 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.184+0000 7f0f91c96640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0f8c0b2cb0 0x7f0f8c142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:50320/0 (socket says 172.21.15.3:50320) 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.184+0000 7f0f91c96640 1 -- 172.21.15.3:0/1365412033 learned_addr learned my addr 172.21.15.3:0/1365412033 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539145265 0 0) 0x7f0f8c0aab60 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.161 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:19:26.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f78003520 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648814150 0 0) 0x7f0f78003520 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.162 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.185+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 >> v1:172.21.15.3:6790/0 conn(0x7f0f8c0b3560 legacy=0x7f0f8c1428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 >> v1:172.21.15.32:6789/0 conn(0x7f0f8c0b2cb0 legacy=0x7f0f8c142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 1 monclient: found mon.noname-a 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c13c380 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f88001690 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f91495640 10 monclient: dump: 2024-03-29T11:19:26.165 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f946fb640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:19:26.166 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.188+0000 7f0f946fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:26.168 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f88001860 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.168 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 1 -- 172.21.15.3:0/1365412033 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f88001cb0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f91495640 10 monclient: dump: 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.169 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f946fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.189+0000 7f0f946fb640 4 set_mon_vals no callback set 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.190+0000 7f0f946fb640 10 monclient: shutdown 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.190+0000 7f0f946fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.170 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.190+0000 7f0f946fb640 1 -- 172.21.15.3:0/1365412033 >> v1:172.21.15.3:6789/0 conn(0x7f0f8c0b28d0 legacy=0x7f0f8c13bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.192+0000 7f0f946fb640 1 -- 172.21.15.3:0/1365412033 shutdown_connections 2024-03-29T11:19:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.193+0000 7f0f946fb640 1 -- 172.21.15.3:0/1365412033 >> 172.21.15.3:0/1365412033 conn(0x7f0f8c066a00 msgr2=0x7f0f8c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.193+0000 7f0f946fb640 1 -- 172.21.15.3:0/1365412033 shutdown_connections 2024-03-29T11:19:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.193+0000 7f0f946fb640 1 -- 172.21.15.3:0/1365412033 wait complete. 2024-03-29T11:19:26.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 10 monclient: build_initial_monmap 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 10 monclient: monmap: 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 librados: starting msgr at 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 librados: starting objecter 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 Processor -- start 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 -- start start 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 librados: setting wanted keys 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 1 librados: calling monclient init 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.194+0000 7f0f946fb640 10 monclient: init 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.195+0000 7f0f946fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.195+0000 7f0f946fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:26.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.198+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0f8c0b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:26.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.198+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-c con 0x7f0f8c13edc0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:26.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.198+0000 7f0f946fb640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8c14d690 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:26.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f946fb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c12f970 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f92497640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0f8c0b28d0 0x7f0f8c1df4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:37411/0 (socket says 172.21.15.3:37411) 2024-03-29T11:19:26.177 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f92497640 1 -- 172.21.15.3:0/3367011957 learned_addr learned my addr 172.21.15.3:0/3367011957 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c12fc50 con 0x7f0f8c14d690 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f8c14da90 con 0x7f0f8c13edc0 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f946fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f946fb640 10 monclient(hunting): authenticate will time out at 1093.453613s 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f82ffd640 10 client.?.objecter ms_handle_connect 0x7f0f8c0b28d0 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.199+0000 7f0f82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926993114 0 0) 0x7f0f8c12f970 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:19:26.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.200+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f70004eb0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376488867 0 0) 0x7f0f70004eb0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f8c12f970 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f88001bb0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 client.?.objecter ms_handle_connect 0x7f0f8c13edc0 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:26.180 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109866167 0 0) 0x7f0f8c14da90 con 0x7f0f8c13edc0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.202+0000 7f0f82ffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f70004eb0 con 0x7f0f8c13edc0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146546574 0 0) 0x7f0f8c12f970 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 >> v1:172.21.15.3:6790/0 conn(0x7f0f8c13edc0 legacy=0x7f0f8c14cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 >> v1:172.21.15.32:6789/0 conn(0x7f0f8c14d690 legacy=0x7f0f8c1e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 monclient: found mon.a 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8c14dc20 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:26.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f88001d10 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f88003be0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f82ffd640 10 monclient: dump: 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:26.184 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f90c94640 4 set_mon_vals no callback set 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f946fb640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f946fb640 10 monclient: _renew_subs 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f946fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.203+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8c1e7cb0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.204+0000 7f0f946fb640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.204+0000 7f0f946fb640 10 monclient: _renew_subs 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.204+0000 7f0f946fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.204+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8c1e8020 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.204+0000 7f0f946fb640 1 librados: init done 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.205+0000 7f0f946fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.205+0000 7f0f946fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.205+0000 7f0f946fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.205+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8c06a400 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.206+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f0f88003d40 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.207+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f0f88084bd0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.207+0000 7f0f82ffd640 10 client.4395.objecter ms_dispatch 0x7f0f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.207+0000 7f0f82ffd640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.207+0000 7f0f82ffd640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:26.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.207+0000 7f0f82ffd640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T11:19:26.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.208+0000 7f0f91c96640 10 monclient: get_auth_request con 0x7f0f7006dd50 auth_method 0 2024-03-29T11:19:26.187 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.208+0000 7f0f82ffd640 10 client.4395.objecter ms_handle_connect 0x7f0f7006dd50 2024-03-29T11:19:26.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.215+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f880504b0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.215+0000 7f0f82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:26.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.215+0000 7f0f82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:26.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.542+0000 7f0f946fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.542+0000 7f0f946fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.542+0000 7f0f946fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:26.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.542+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f8c0b5ac0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.544+0000 7f0f82ffd640 1 -- 172.21.15.3:0/3367011957 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f0f880523e0 con 0x7f0f8c0b28d0 2024-03-29T11:19:26.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.544+0000 7f0f82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:26.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.544+0000 7f0f82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:26.521 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:26.521 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 10 librados: watch_flush enter 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 10 librados: watch_flush exit 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 10 client.4395.objecter successfully canceled tick 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 >> v1:172.21.15.3:6816/844689376 conn(0x7f0f7006dd50 legacy=0x7f0f70070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 10 monclient: shutdown 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 >> v1:172.21.15.3:6789/0 conn(0x7f0f8c0b28d0 legacy=0x7f0f8c1df4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 shutdown_connections 2024-03-29T11:19:26.525 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 >> 172.21.15.3:0/3367011957 conn(0x7f0f8c066a00 msgr2=0x7f0f8c141870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:26.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 shutdown_connections 2024-03-29T11:19:26.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 -- 172.21.15.3:0/3367011957 wait complete. 2024-03-29T11:19:26.526 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:26.548+0000 7f0f946fb640 1 librados: shutdown 2024-03-29T11:19:26.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:33.348 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.520+0000 7f61c704e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.520+0000 7f61c704e640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.520+0000 7f61c704e640 10 monclient: monmap: 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:33.521697+0000 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:33.521697+0000 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.498 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.522+0000 7f61c704e640 1 Processor -- start 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.522+0000 7f61c704e640 1 -- start start 2024-03-29T11:19:33.499 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.522+0000 7f61c704e640 10 monclient: init 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-a con 0x7f61c00a9d00 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-c con 0x7f61c00aaef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-b con 0x7f61c00af110 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0141e60 con 0x7f61c00a9d00 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c00b0090 con 0x7f61c00aaef0 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0141a30 con 0x7f61c00af110 2024-03-29T11:19:33.501 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.502 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.523+0000 7f61c704e640 10 monclient(hunting): authenticate will time out at 1100.777907s 2024-03-29T11:19:33.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61c4dea640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f61c00a9d00 0x7f61c00af980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51201/0 (socket says 172.21.15.3:51201) 2024-03-29T11:19:33.503 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61c4dea640 1 -- 172.21.15.3:0/97815126 learned_addr learned my addr 172.21.15.3:0/97815126 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:33.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060350545 0 0) 0x7f61c0141e60 con 0x7f61c00a9d00 2024-03-29T11:19:33.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.504 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.527+0000 7f61bf7fe640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:19:33.511 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.528+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a8003db0 con 0x7f61c00a9d00 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202967281 0 0) 0x7f61c0141a30 con 0x7f61c00af110 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.530+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a8003520 con 0x7f61c00af110 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124175748 0 0) 0x7f61c00b0090 con 0x7f61c00aaef0 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:19:33.512 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0141a30 con 0x7f61c00aaef0 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1396388020 0 0) 0x7f61a8003db0 con 0x7f61c00a9d00 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 >> v1:172.21.15.3:6790/0 conn(0x7f61c00af110 legacy=0x7f61c01498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 >> v1:172.21.15.32:6789/0 conn(0x7f61c00aaef0 legacy=0x7f61c00aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 monclient: found mon.noname-a 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c01422f0 con 0x7f61c00a9d00 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f61b0001670 con 0x7f61c00a9d00 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.531+0000 7f61bf7fe640 10 monclient: dump: 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61c704e640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61c704e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:33.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b0001850 con 0x7f61c00a9d00 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 1 -- 172.21.15.3:0/97815126 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f61b0001670 con 0x7f61c00a9d00 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61c704e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61c704e640 4 set_mon_vals no callback set 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.532+0000 7f61bf7fe640 10 monclient: dump: 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:33.516 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.533+0000 7f61c704e640 10 monclient: shutdown 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.533+0000 7f61c704e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.533+0000 7f61c704e640 1 -- 172.21.15.3:0/97815126 >> v1:172.21.15.3:6789/0 conn(0x7f61c00a9d00 legacy=0x7f61c00af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.533+0000 7f61c704e640 1 -- 172.21.15.3:0/97815126 shutdown_connections 2024-03-29T11:19:33.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.533+0000 7f61c704e640 1 -- 172.21.15.3:0/97815126 >> 172.21.15.3:0/97815126 conn(0x7f61c00b3c10 msgr2=0x7f61c00b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.537+0000 7f61c704e640 1 -- 172.21.15.3:0/97815126 shutdown_connections 2024-03-29T11:19:33.518 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.537+0000 7f61c704e640 1 -- 172.21.15.3:0/97815126 wait complete. 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.538+0000 7f61c704e640 10 monclient: build_initial_monmap 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.538+0000 7f61c704e640 10 monclient: monmap: 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:33.519 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.538+0000 7f61c704e640 1 librados: starting msgr at 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.538+0000 7f61c704e640 1 librados: starting objecter 2024-03-29T11:19:33.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.538+0000 7f61c704e640 1 Processor -- start 2024-03-29T11:19:33.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.540+0000 7f61c704e640 1 -- start start 2024-03-29T11:19:33.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.540+0000 7f61c704e640 1 librados: setting wanted keys 2024-03-29T11:19:33.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.540+0000 7f61c704e640 1 librados: calling monclient init 2024-03-29T11:19:33.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.540+0000 7f61c704e640 10 monclient: init 2024-03-29T11:19:33.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-b con 0x7f61c00aaef0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-c con 0x7f61c00af110 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient(hunting): picked mon.noname-a con 0x7f61c013e300 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0072b60 con 0x7f61c013e300 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0070540 con 0x7f61c00aaef0 2024-03-29T11:19:33.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c0071640 con 0x7f61c00af110 2024-03-29T11:19:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:33.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.541+0000 7f61c704e640 10 monclient(hunting): authenticate will time out at 1100.795585s 2024-03-29T11:19:33.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.546+0000 7f61c55eb640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f61c013e300 0x7f61c01e6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51595/0 (socket says 172.21.15.3:51595) 2024-03-29T11:19:33.527 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.546+0000 7f61c55eb640 1 -- 172.21.15.3:0/2139602114 learned_addr learned my addr 172.21.15.3:0/2139602114 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter ms_handle_connect 0x7f61c013e300 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711488733 0 0) 0x7f61c0072b60 con 0x7f61c013e300 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:19:33.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ac004eb0 con 0x7f61c013e300 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter ms_handle_connect 0x7f61c00aaef0 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter ms_handle_connect 0x7f61c00af110 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852947443 0 0) 0x7f61c0070540 con 0x7f61c00aaef0 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.547+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0072b60 con 0x7f61c00aaef0 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833145795 0 0) 0x7f61c0071640 con 0x7f61c00af110 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:33.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c0070540 con 0x7f61c00af110 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 708561947 0 0) 0x7f61ac004eb0 con 0x7f61c013e300 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c0071640 con 0x7f61c013e300 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f61b8001cb0 con 0x7f61c013e300 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.531 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1488725150 0 0) 0x7f61c0072b60 con 0x7f61c00aaef0 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61ac004eb0 con 0x7f61c00aaef0 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f61b0001c20 con 0x7f61c00aaef0 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:33.532 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154678726 0 0) 0x7f61c0070540 con 0x7f61c00af110 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61c0072b60 con 0x7f61c00af110 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668892955 0 0) 0x7f61c0071640 con 0x7f61c013e300 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 >> v1:172.21.15.3:6790/0 conn(0x7f61c00af110 legacy=0x7f61c0072450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 >> v1:172.21.15.32:6789/0 conn(0x7f61c00aaef0 legacy=0x7f61c0071d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:19:33.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 monclient: found mon.a 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c006f2c0 con 0x7f61c013e300 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 10 monclient: _renew_subs 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c006fe20 con 0x7f61c013e300 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 10 monclient: _renew_subs 2024-03-29T11:19:33.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c01e7d20 con 0x7f61c013e300 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61c704e640 1 librados: init done 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61b8003970 con 0x7f61c013e300 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f61b8003cc0 con 0x7f61c013e300 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.548+0000 7f61bd7fa640 10 monclient: dump: 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:33.535 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:33.536 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:33.536 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:33.536 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:33.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.549+0000 7f61be7fc640 4 set_mon_vals no callback set 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.551+0000 7f61c704e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.551+0000 7f61c704e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.551+0000 7f61c704e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.551+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a0005080 con 0x7f61c013e300 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.553+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f61b8003e20 con 0x7f61c013e300 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61c4dea640 10 monclient: get_auth_request con 0x7f61ac06e040 auth_method 0 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 10 client.4401.objecter ms_handle_connect 0x7f61ac06e040 2024-03-29T11:19:33.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f61b8084ce0 con 0x7f61c013e300 2024-03-29T11:19:33.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 10 client.4401.objecter ms_dispatch 0x7f61c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:33.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:33.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:33.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.554+0000 7f61bd7fa640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:19:33.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.562+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61b80505c0 con 0x7f61c013e300 2024-03-29T11:19:33.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.562+0000 7f61bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:33.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.562+0000 7f61bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.889+0000 7f61c704e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.889+0000 7f61c704e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.889+0000 7f61c704e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.889+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61a0005330 con 0x7f61c013e300 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.890+0000 7f61bd7fa640 1 -- 172.21.15.3:0/2139602114 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f61b80524f0 con 0x7f61c013e300 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.890+0000 7f61bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:33.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.890+0000 7f61bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:33.869 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:33.869 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.895+0000 7f61c704e640 10 librados: watch_flush enter 2024-03-29T11:19:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 10 librados: watch_flush exit 2024-03-29T11:19:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:19:33.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:19:33.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 >> v1:172.21.15.3:6816/844689376 conn(0x7f61ac06e040 legacy=0x7f61ac070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 10 monclient: shutdown 2024-03-29T11:19:33.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:33.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 >> v1:172.21.15.3:6789/0 conn(0x7f61c013e300 legacy=0x7f61c01e6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 shutdown_connections 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 >> 172.21.15.3:0/2139602114 conn(0x7f61c00b3c10 msgr2=0x7f61c00b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 shutdown_connections 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.896+0000 7f61c704e640 1 -- 172.21.15.3:0/2139602114 wait complete. 2024-03-29T11:19:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:33.897+0000 7f61c704e640 1 librados: shutdown 2024-03-29T11:19:33.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:40.696 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.868+0000 7f955c159640 10 monclient: get_monmap_and_config 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.868+0000 7f955c159640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.868+0000 7f955c159640 10 monclient: monmap: 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:40.869502+0000 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:40.869502+0000 2024-03-29T11:19:40.845 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.846 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.869+0000 7f955c159640 1 Processor -- start 2024-03-29T11:19:40.847 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.869+0000 7f955c159640 1 -- start start 2024-03-29T11:19:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.869+0000 7f955c159640 10 monclient: init 2024-03-29T11:19:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.870+0000 7f955c159640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.870+0000 7f955c159640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:40.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 10 monclient(hunting): picked mon.noname-a con 0x7f95540b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 10 monclient(hunting): picked mon.noname-b con 0x7f95540b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 10 monclient(hunting): picked mon.noname-c con 0x7f95540b3560 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540aab60 con 0x7f95540b28d0 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540aae40 con 0x7f95540b3560 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540ab120 con 0x7f95540b2cb0 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.872+0000 7f955c159640 10 monclient(hunting): authenticate will time out at 1108.126280s 2024-03-29T11:19:40.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.875+0000 7f955a6f6640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f95540b3560 0x7f95541428a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:55421/0 (socket says 172.21.15.3:55421) 2024-03-29T11:19:40.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.875+0000 7f955a6f6640 1 -- 172.21.15.3:0/4142166062 learned_addr learned my addr 172.21.15.3:0/4142166062 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:40.852 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.875+0000 7f9559ef5640 1 --1- 172.21.15.3:0/4142166062 >> v1:172.21.15.3:6789/0 conn(0x7f95540b28d0 0x7f955413bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38645/0 (socket says 172.21.15.3:38645) 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.875+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110024334 0 0) 0x7f95540aae40 con 0x7f95540b3560 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9540003520 con 0x7f95540b3560 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782407254 0 0) 0x7f95540aab60 con 0x7f95540b28d0 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.853 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:19:40.854 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95540aae40 con 0x7f95540b28d0 2024-03-29T11:19:40.856 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972476168 0 0) 0x7f9540003520 con 0x7f95540b3560 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 >> v1:172.21.15.3:6790/0 conn(0x7f95540b2cb0 legacy=0x7f9554142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 >> v1:172.21.15.3:6789/0 conn(0x7f95540b28d0 legacy=0x7f955413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 monclient: found mon.noname-c 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955413c380 con 0x7f95540b3560 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f954c001a10 con 0x7f95540b3560 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.857 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f9558ef3640 10 monclient: dump: 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f955c159640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.876+0000 7f955c159640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:40.858 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f954c001b70 con 0x7f95540b3560 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 1 -- 172.21.15.3:0/4142166062 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f954c001fc0 con 0x7f95540b3560 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f955c159640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f955c159640 4 set_mon_vals no callback set 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.877+0000 7f9558ef3640 10 monclient: dump: 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:40.859 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.878+0000 7f955c159640 10 monclient: shutdown 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.878+0000 7f955c159640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.878+0000 7f955c159640 1 -- 172.21.15.3:0/4142166062 >> v1:172.21.15.32:6789/0 conn(0x7f95540b3560 legacy=0x7f95541428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.880+0000 7f955c159640 1 -- 172.21.15.3:0/4142166062 shutdown_connections 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.880+0000 7f955c159640 1 -- 172.21.15.3:0/4142166062 >> 172.21.15.3:0/4142166062 conn(0x7f9554066a00 msgr2=0x7f9554066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:40.860 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.881+0000 7f955c159640 1 -- 172.21.15.3:0/4142166062 shutdown_connections 2024-03-29T11:19:40.861 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.884+0000 7f955c159640 1 -- 172.21.15.3:0/4142166062 wait complete. 2024-03-29T11:19:40.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.885+0000 7f955c159640 10 monclient: build_initial_monmap 2024-03-29T11:19:40.862 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.885+0000 7f955c159640 10 monclient: monmap: 2024-03-29T11:19:40.862 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:40.862 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.885+0000 7f955c159640 1 librados: starting msgr at 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.885+0000 7f955c159640 1 librados: starting objecter 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.885+0000 7f955c159640 1 Processor -- start 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.886+0000 7f955c159640 1 -- start start 2024-03-29T11:19:40.863 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.886+0000 7f955c159640 1 librados: setting wanted keys 2024-03-29T11:19:40.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.886+0000 7f955c159640 1 librados: calling monclient init 2024-03-29T11:19:40.864 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.886+0000 7f955c159640 10 monclient: init 2024-03-29T11:19:40.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.886+0000 7f955c159640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:40.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.887+0000 7f955c159640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:40.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.887+0000 7f955c159640 10 monclient(hunting): picked mon.noname-b con 0x7f95540b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:40.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.887+0000 7f955c159640 10 monclient(hunting): picked mon.noname-c con 0x7f955406f7e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:40.865 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 10 monclient(hunting): picked mon.noname-a con 0x7f955406de80 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955406e290 con 0x7f955406de80 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955413dd50 con 0x7f95540b28d0 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955406f0e0 con 0x7f955406f7e0 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.889+0000 7f955c159640 10 monclient(hunting): authenticate will time out at 1108.143128s 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.891+0000 7f955a6f6640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f955406de80 0x7f95541eed30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38899/0 (socket says 172.21.15.3:38899) 2024-03-29T11:19:40.868 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.891+0000 7f955a6f6640 1 -- 172.21.15.3:0/3807825294 learned_addr learned my addr 172.21.15.3:0/3807825294 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:40.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.894+0000 7f954a7fc640 10 client.?.objecter ms_handle_connect 0x7f955406f7e0 2024-03-29T11:19:40.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.894+0000 7f954a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201841834 0 0) 0x7f955406f0e0 con 0x7f955406f7e0 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:19:40.872 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.895+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9538004eb0 con 0x7f955406f7e0 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 client.?.objecter ms_handle_connect 0x7f955406de80 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603031277 0 0) 0x7f955406e290 con 0x7f955406de80 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:19:40.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.897+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f955406f0e0 con 0x7f955406de80 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 client.?.objecter ms_handle_connect 0x7f95540b28d0 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052029190 0 0) 0x7f955406f0e0 con 0x7f955406de80 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f955406e290 con 0x7f955406de80 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 368792752 0 0) 0x7f9538004eb0 con 0x7f955406f7e0 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f955406f0e0 con 0x7f955406f7e0 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f954c001ec0 con 0x7f955406de80 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:40.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9544001840 con 0x7f955406f7e0 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:40.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient(hunting): dump: 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1692226882 0 0) 0x7f955406f0e0 con 0x7f955406f7e0 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 >> v1:172.21.15.32:6789/0 conn(0x7f95540b28d0 legacy=0x7f955413e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 >> v1:172.21.15.3:6789/0 conn(0x7f955406de80 legacy=0x7f95541eed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 monclient: found mon.c 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95541325f0 con 0x7f955406f7e0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.898+0000 7f954a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 10 monclient: _renew_subs 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9554131a00 con 0x7f955406f7e0 2024-03-29T11:19:40.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:40.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 10 monclient: _renew_subs 2024-03-29T11:19:40.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:40.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f955413efc0 con 0x7f955406f7e0 2024-03-29T11:19:40.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.899+0000 7f955c159640 1 librados: init done 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9544001d70 con 0x7f955406f7e0 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f955c159640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f955c159640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f955c159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95541327e0 con 0x7f955406f7e0 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9544003790 con 0x7f955406f7e0 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 10 monclient: dump: 2024-03-29T11:19:40.882 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f95440197a0 con 0x7f955406f7e0 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.902+0000 7f954bfff640 4 set_mon_vals no callback set 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.903+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f9544083820 con 0x7f955406f7e0 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.903+0000 7f9559ef5640 10 monclient: get_auth_request con 0x7f953806e060 auth_method 0 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.904+0000 7f954a7fc640 10 client.4427.objecter ms_dispatch 0x7f9554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:40.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.904+0000 7f954a7fc640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:40.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.904+0000 7f954a7fc640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:40.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.904+0000 7f954a7fc640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:19:40.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.904+0000 7f954a7fc640 10 client.4427.objecter ms_handle_connect 0x7f953806e060 2024-03-29T11:19:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.911+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f954404efb0 con 0x7f955406f7e0 2024-03-29T11:19:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.911+0000 7f954a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:40.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:40.911+0000 7f954a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.236+0000 7f955c159640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.236+0000 7f955c159640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.236+0000 7f955c159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:41.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.236+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f955406a400 con 0x7f955406f7e0 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:41.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.239+0000 7f954a7fc640 1 -- 172.21.15.3:0/3807825294 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f95440510b0 con 0x7f955406f7e0 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.239+0000 7f954a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:41.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.239+0000 7f954a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 10 librados: watch_flush enter 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 10 librados: watch_flush exit 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 >> v1:172.21.15.3:6816/844689376 conn(0x7f953806e060 legacy=0x7f9538070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 10 monclient: shutdown 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.243+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 >> v1:172.21.15.3:6790/0 conn(0x7f955406f7e0 legacy=0x7f955413d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.244+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 shutdown_connections 2024-03-29T11:19:41.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.244+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 >> 172.21.15.3:0/3807825294 conn(0x7f9554066a00 msgr2=0x7f9554134dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:41.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.245+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 shutdown_connections 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.248+0000 7f955c159640 1 -- 172.21.15.3:0/3807825294 wait complete. 2024-03-29T11:19:41.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:41.248+0000 7f955c159640 1 librados: shutdown 2024-03-29T11:19:41.243 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:48.049 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.226+0000 7f9a48e99640 10 monclient: get_monmap_and_config 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.226+0000 7f9a48e99640 10 monclient: build_initial_monmap 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.226+0000 7f9a48e99640 10 monclient: monmap: 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:48.227649+0000 2024-03-29T11:19:48.203 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:48.227649+0000 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.228+0000 7f9a48e99640 1 Processor -- start 2024-03-29T11:19:48.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.228+0000 7f9a48e99640 1 -- start start 2024-03-29T11:19:48.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.228+0000 7f9a48e99640 10 monclient: init 2024-03-29T11:19:48.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:48.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-b con 0x7f9a440b4660 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-c con 0x7f9a440a9ee0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-a con 0x7f9a44142d00 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a440aac10 con 0x7f9a44142d00 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a44145590 con 0x7f9a440a9ee0 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a44142010 con 0x7f9a440b4660 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:48.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.229+0000 7f9a48e99640 10 monclient(hunting): authenticate will time out at 1115.483093s 2024-03-29T11:19:48.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.234+0000 7f9a42ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9a440a9ee0 0x7f9a44145a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:44691/0 (socket says 172.21.15.3:44691) 2024-03-29T11:19:48.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.234+0000 7f9a42ffd640 1 -- 172.21.15.3:0/3021938276 learned_addr learned my addr 172.21.15.3:0/3021938276 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:48.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940724502 0 0) 0x7f9a44142010 con 0x7f9a440b4660 2024-03-29T11:19:48.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 10 monclient(hunting): _negotiate 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.235+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a30003520 con 0x7f9a440b4660 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271893156 0 0) 0x7f9a30003520 con 0x7f9a440b4660 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 >> v1:172.21.15.32:6789/0 conn(0x7f9a440a9ee0 legacy=0x7f9a44145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 >> v1:172.21.15.3:6789/0 conn(0x7f9a44142d00 legacy=0x7f9a4414d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: _finish_hunting 0 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 monclient: found mon.noname-b 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a440aae40 con 0x7f9a440b4660 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: _finish_auth 0 2024-03-29T11:19:48.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: _check_auth_tickets 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9a2c001690 con 0x7f9a440b4660 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: dump: 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:48.216 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a48e99640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a48e99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a2c0018b0 con 0x7f9a440b4660 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 1 -- 172.21.15.3:0/3021938276 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9a2c001cc0 con 0x7f9a440b4660 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a427fc640 10 monclient: dump: 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.217 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a48e99640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.236+0000 7f9a48e99640 4 set_mon_vals no callback set 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 10 monclient: shutdown 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 1 -- 172.21.15.3:0/3021938276 >> v1:172.21.15.3:6790/0 conn(0x7f9a440b4660 legacy=0x7f9a441425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 1 -- 172.21.15.3:0/3021938276 shutdown_connections 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 1 -- 172.21.15.3:0/3021938276 >> 172.21.15.3:0/3021938276 conn(0x7f9a44066a00 msgr2=0x7f9a44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 1 -- 172.21.15.3:0/3021938276 shutdown_connections 2024-03-29T11:19:48.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.237+0000 7f9a48e99640 1 -- 172.21.15.3:0/3021938276 wait complete. 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.238+0000 7f9a48e99640 10 monclient: build_initial_monmap 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.238+0000 7f9a48e99640 10 monclient: monmap: 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:48.219 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 librados: starting msgr at 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 librados: starting objecter 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 Processor -- start 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 -- start start 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 librados: setting wanted keys 2024-03-29T11:19:48.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 1 librados: calling monclient init 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.239+0000 7f9a48e99640 10 monclient: init 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.240+0000 7f9a48e99640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.240+0000 7f9a48e99640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-b con 0x7f9a440a9ee0 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-c con 0x7f9a44142d00 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 10 monclient(hunting): picked mon.noname-a con 0x7f9a4413aea0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a441d7600 con 0x7f9a4413aea0 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a440b1840 con 0x7f9a440a9ee0 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a440ae2d0 con 0x7f9a44142d00 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:48.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.241+0000 7f9a48e99640 10 monclient(hunting): authenticate will time out at 1115.495726s 2024-03-29T11:19:48.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.246+0000 7f9a42ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f9a44142d00 0x7f9a441d7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53803/0 (socket says 172.21.15.3:53803) 2024-03-29T11:19:48.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.246+0000 7f9a42ffd640 1 -- 172.21.15.3:0/861669218 learned_addr learned my addr 172.21.15.3:0/861669218 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:48.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.246+0000 7f9a3bfff640 10 client.?.objecter ms_handle_connect 0x7f9a44142d00 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.246+0000 7f9a3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 client.?.objecter ms_handle_connect 0x7f9a4413aea0 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143410825 0 0) 0x7f9a440ae2d0 con 0x7f9a44142d00 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a24004eb0 con 0x7f9a44142d00 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274416060 0 0) 0x7f9a441d7600 con 0x7f9a4413aea0 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:48.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.247+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a240051d0 con 0x7f9a4413aea0 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1631939569 0 0) 0x7f9a24004eb0 con 0x7f9a44142d00 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a24005640 con 0x7f9a44142d00 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9a34001930 con 0x7f9a44142d00 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.228 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3856305902 0 0) 0x7f9a240051d0 con 0x7f9a4413aea0 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a24004eb0 con 0x7f9a4413aea0 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9a3c001a70 con 0x7f9a4413aea0 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient(hunting): dump: 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.229 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1264944792 0 0) 0x7f9a24005640 con 0x7f9a44142d00 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 >> v1:172.21.15.32:6789/0 conn(0x7f9a440a9ee0 legacy=0x7f9a440b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 >> v1:172.21.15.3:6789/0 conn(0x7f9a4413aea0 legacy=0x7f9a441e20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 monclient: found mon.c 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a4413b940 con 0x7f9a44142d00 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:19:48.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 10 monclient: _renew_subs 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a441d81a0 con 0x7f9a44142d00 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a340047e0 con 0x7f9a44142d00 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9a34004bf0 con 0x7f9a44142d00 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a3bfff640 10 monclient: dump: 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:48.231 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 10 monclient: _renew_subs 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a441e37d0 con 0x7f9a44142d00 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.248+0000 7f9a48e99640 1 librados: init done 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.250+0000 7f9a48e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.250+0000 7f9a48e99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.250+0000 7f9a48e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.250+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a440b0800 con 0x7f9a44142d00 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.250+0000 7f9a417fa640 4 set_mon_vals no callback set 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.254+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f9a34004dd0 con 0x7f9a44142d00 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.255+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f9a34084780 con 0x7f9a44142d00 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.255+0000 7f9a3bfff640 10 client.4433.objecter ms_dispatch 0x7f9a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.255+0000 7f9a3bfff640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.255+0000 7f9a3bfff640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:48.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.255+0000 7f9a3bfff640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:19:48.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.256+0000 7f9a437fe640 10 monclient: get_auth_request con 0x7f9a2406e6d0 auth_method 0 2024-03-29T11:19:48.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.257+0000 7f9a3bfff640 10 client.4433.objecter ms_handle_connect 0x7f9a2406e6d0 2024-03-29T11:19:48.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.262+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a34050060 con 0x7f9a44142d00 2024-03-29T11:19:48.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.262+0000 7f9a3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:48.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.262+0000 7f9a3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:48.562 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.586+0000 7f9a48e99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.586+0000 7f9a48e99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.586+0000 7f9a48e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:19:48.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.586+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a440b0800 con 0x7f9a44142d00 2024-03-29T11:19:48.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.587+0000 7f9a3bfff640 1 -- 172.21.15.3:0/861669218 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f9a440b0800 con 0x7f9a44142d00 2024-03-29T11:19:48.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.587+0000 7f9a3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:48.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.587+0000 7f9a3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:48.567 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:48.567 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:48.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.591+0000 7f9a48e99640 10 librados: watch_flush enter 2024-03-29T11:19:48.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.592+0000 7f9a48e99640 10 librados: watch_flush exit 2024-03-29T11:19:48.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.593+0000 7f9a48e99640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:19:48.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.593+0000 7f9a48e99640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 >> v1:172.21.15.3:6816/844689376 conn(0x7f9a2406e6d0 legacy=0x7f9a24070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 10 monclient: shutdown 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 >> v1:172.21.15.3:6790/0 conn(0x7f9a44142d00 legacy=0x7f9a441d7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 shutdown_connections 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 >> 172.21.15.3:0/861669218 conn(0x7f9a44066a00 msgr2=0x7f9a440b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 shutdown_connections 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 -- 172.21.15.3:0/861669218 wait complete. 2024-03-29T11:19:48.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:48.595+0000 7f9a48e99640 1 librados: shutdown 2024-03-29T11:19:48.590 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:55.397 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.573+0000 7fc7e359e640 10 monclient: get_monmap_and_config 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.573+0000 7fc7e359e640 10 monclient: build_initial_monmap 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.573+0000 7fc7e359e640 10 monclient: monmap: 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:19:55.574000+0000 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:19:55.574000+0000 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:55.552 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:19:55.553 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:19:55.553 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.575+0000 7fc7e359e640 1 Processor -- start 2024-03-29T11:19:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.576+0000 7fc7e359e640 1 -- start start 2024-03-29T11:19:55.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.576+0000 7fc7e359e640 10 monclient: init 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc034850 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc0c9550 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc02aac0 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc02c440 con 0x7fc7dc034850 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc02aed0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc02b190 con 0x7fc7dc02aac0 2024-03-29T11:19:55.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.577+0000 7fc7e359e640 10 monclient(hunting): authenticate will time out at 1122.831388s 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.583+0000 7fc7e259c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc7dc034850 0x7fc7dc0c8e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47651/0 (socket says 172.21.15.3:47651) 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.583+0000 7fc7e259c640 1 -- 172.21.15.3:0/572467098 learned_addr learned my addr 172.21.15.3:0/572467098 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527736353 0 0) 0x7fc7dc02c440 con 0x7fc7dc034850 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7cc003520 con 0x7fc7dc034850 2024-03-29T11:19:55.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 795919876 0 0) 0x7fc7cc003520 con 0x7fc7dc034850 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 >> v1:172.21.15.3:6790/0 conn(0x7fc7dc02aac0 legacy=0x7fc7dc0cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 >> v1:172.21.15.32:6789/0 conn(0x7fc7dc0c9550 legacy=0x7fc7dc0cbea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 monclient: found mon.noname-a 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc033060 con 0x7fc7dc034850 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.585+0000 7fc7e159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e359e640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc7e404f730 con 0x7fc7dc034850 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:55.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e159a640 10 monclient: dump: 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.586+0000 7fc7e359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e40008d0 con 0x7fc7dc034850 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 1 -- 172.21.15.3:0/572467098 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc7e404f080 con 0x7fc7dc034850 2024-03-29T11:19:55.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e159a640 10 monclient: dump: 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:55.567 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:55.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.587+0000 7fc7e359e640 4 set_mon_vals no callback set 2024-03-29T11:19:55.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.590+0000 7fc7e359e640 10 monclient: shutdown 2024-03-29T11:19:55.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.590+0000 7fc7e359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:55.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.590+0000 7fc7e359e640 1 -- 172.21.15.3:0/572467098 >> v1:172.21.15.3:6789/0 conn(0x7fc7dc034850 legacy=0x7fc7dc0c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.591+0000 7fc7e359e640 1 -- 172.21.15.3:0/572467098 shutdown_connections 2024-03-29T11:19:55.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.591+0000 7fc7e359e640 1 -- 172.21.15.3:0/572467098 >> 172.21.15.3:0/572467098 conn(0x7fc7dc0058f0 msgr2=0x7fc7dc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:55.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.591+0000 7fc7e359e640 1 -- 172.21.15.3:0/572467098 shutdown_connections 2024-03-29T11:19:55.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.594+0000 7fc7e359e640 1 -- 172.21.15.3:0/572467098 wait complete. 2024-03-29T11:19:55.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.595+0000 7fc7e359e640 10 monclient: build_initial_monmap 2024-03-29T11:19:55.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.595+0000 7fc7e359e640 10 monclient: monmap: 2024-03-29T11:19:55.571 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:19:55.572 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.595+0000 7fc7e359e640 1 librados: starting msgr at 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.595+0000 7fc7e359e640 1 librados: starting objecter 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.595+0000 7fc7e359e640 1 Processor -- start 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 1 -- start start 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 1 librados: setting wanted keys 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 1 librados: calling monclient init 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 10 monclient: init 2024-03-29T11:19:55.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:55.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:55.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc02aac0 addr v1:172.21.15.3:6789/0 2024-03-29T11:19:55.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.596+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc034850 addr v1:172.21.15.3:6790/0 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.599+0000 7fc7e359e640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc0c9550 addr v1:172.21.15.32:6789/0 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e359e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0c4f10 con 0x7fc7dc02aac0 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e359e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0c51f0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e359e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7dc0c54d0 con 0x7fc7dc034850 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e359e640 10 monclient(hunting): authenticate will time out at 1122.854289s 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e2d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc7dc0c9550 0x7fc7dc02c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:50651/0 (socket says 172.21.15.3:50651) 2024-03-29T11:19:55.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7e2d9d640 1 -- 172.21.15.3:0/3117869562 learned_addr learned my addr 172.21.15.3:0/3117869562 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7dc0c9550 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358019139 0 0) 0x7fc7dc0c51f0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:19:55.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.600+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c4004eb0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 445791678 0 0) 0x7fc7c4004eb0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7dc0c51f0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc7d4004000 con 0x7fc7dc0c9550 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 10 monclient(hunting): dump: 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.601+0000 7fc7d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.602+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1368290487 0 0) 0x7fc7dc0c51f0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.602+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 >> v1:172.21.15.3:6790/0 conn(0x7fc7dc034850 legacy=0x7fc7dc0c1aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 >> v1:172.21.15.3:6789/0 conn(0x7fc7dc02aac0 legacy=0x7fc7dc0c3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 1 monclient: found mon.b 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0c40e0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:55.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 client.?.objecter ms_handle_connect 0x7fc7dc02aac0 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 10 monclient: _renew_subs 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc0c2e70 con 0x7fc7dc0c9550 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 10 monclient: _renew_subs 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.605+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc16cc00 con 0x7fc7dc0c9550 2024-03-29T11:19:55.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.606+0000 7fc7e359e640 1 librados: init done 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7d4004160 con 0x7fc7dc0c9550 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc7d4001170 con 0x7fc7dc0c9550 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 10 monclient: dump: 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:55.587 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.608+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fc7d4001390 con 0x7fc7dc0c9550 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fc7d4084b10 con 0x7fc7dc0c9550 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7d2ffd640 10 client.4426.objecter ms_dispatch 0x7fc7e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7d2ffd640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7d2ffd640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7d2ffd640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7e0d99640 4 set_mon_vals no callback set 2024-03-29T11:19:55.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.609+0000 7fc7e359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.612+0000 7fc7e359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.612+0000 7fc7e359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:55.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.612+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc004cd0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.612+0000 7fc7e259c640 10 monclient: get_auth_request con 0x7fc7c406dc40 auth_method 0 2024-03-29T11:19:55.589 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.612+0000 7fc7d2ffd640 10 client.4426.objecter ms_handle_connect 0x7fc7c406dc40 2024-03-29T11:19:55.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.624+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7d4050470 con 0x7fc7dc0c9550 2024-03-29T11:19:55.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.624+0000 7fc7d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:55.601 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.624+0000 7fc7d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:55.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.956+0000 7fc7e359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:55.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.956+0000 7fc7e359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:55.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.956+0000 7fc7e359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:19:55.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.956+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7dc0c9080 con 0x7fc7dc0c9550 2024-03-29T11:19:55.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.957+0000 7fc7d2ffd640 1 -- 172.21.15.3:0/3117869562 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fc7d40523a0 con 0x7fc7dc0c9550 2024-03-29T11:19:55.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.957+0000 7fc7d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:55.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.957+0000 7fc7d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:55.937 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:19:55.937 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:55.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.961+0000 7fc7e359e640 10 librados: watch_flush enter 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.961+0000 7fc7e359e640 10 librados: watch_flush exit 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.961+0000 7fc7e359e640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.961+0000 7fc7e359e640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 >> v1:172.21.15.3:6816/844689376 conn(0x7fc7c406dc40 legacy=0x7fc7c4070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 10 monclient: shutdown 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 >> v1:172.21.15.32:6789/0 conn(0x7fc7dc0c9550 legacy=0x7fc7dc02c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 shutdown_connections 2024-03-29T11:19:55.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.962+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 >> 172.21.15.3:0/3117869562 conn(0x7fc7dc0058f0 msgr2=0x7fc7dc0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:55.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.963+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 shutdown_connections 2024-03-29T11:19:55.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.963+0000 7fc7e359e640 1 -- 172.21.15.3:0/3117869562 wait complete. 2024-03-29T11:19:55.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:19:55.963+0000 7fc7e359e640 1 librados: shutdown 2024-03-29T11:19:55.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:02.764 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.926+0000 7fb05480d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.926+0000 7fb05480d640 10 monclient: build_initial_monmap 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.926+0000 7fb05480d640 10 monclient: monmap: 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:02.927301+0000 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:02.927301+0000 2024-03-29T11:20:02.903 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:02.904 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.904 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:02.904 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:02.904 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:02.904 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.927+0000 7fb05480d640 1 Processor -- start 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.927+0000 7fb05480d640 1 -- start start 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.927+0000 7fb05480d640 10 monclient: init 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-c con 0x7fb0500b1f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-a con 0x7fb050146400 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-b con 0x7fb05014fd10 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05006ed20 con 0x7fb050146400 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05006ef90 con 0x7fb0500b1f20 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500b42e0 con 0x7fb05014fd10 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:02.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb05480d640 10 monclient(hunting): authenticate will time out at 1130.182441s 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb04e59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb0500b1f20 0x7fb0501434f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:54254/0 (socket says 172.21.15.3:54254) 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.928+0000 7fb04e59c640 1 -- 172.21.15.3:0/3682883764 learned_addr learned my addr 172.21.15.3:0/3682883764 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206635114 0 0) 0x7fb05006ef90 con 0x7fb0500b1f20 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb038003520 con 0x7fb0500b1f20 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023869459 0 0) 0x7fb0500b42e0 con 0x7fb05014fd10 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:20:02.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05006ef90 con 0x7fb05014fd10 2024-03-29T11:20:02.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2870504661 0 0) 0x7fb038003520 con 0x7fb0500b1f20 2024-03-29T11:20:02.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 >> v1:172.21.15.3:6790/0 conn(0x7fb05014fd10 legacy=0x7fb050152100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.929+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 >> v1:172.21.15.3:6789/0 conn(0x7fb050146400 legacy=0x7fb050146880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 1 monclient: found mon.noname-c 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0500b3690 con 0x7fb0500b1f20 2024-03-29T11:20:02.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: _finish_auth 0 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb05480d640 5 monclient: authenticate success, global_id 4429 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb05480d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb044001690 con 0x7fb0500b1f20 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb04d59a640 10 monclient: dump: 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.911 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.930+0000 7fb05480d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0440018b0 con 0x7fb0500b1f20 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb05480d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb05480d640 4 set_mon_vals no callback set 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 1 -- 172.21.15.3:0/3682883764 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb044001c40 con 0x7fb0500b1f20 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:02.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.932+0000 7fb04d59a640 10 monclient: dump: 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 10 monclient: shutdown 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 1 -- 172.21.15.3:0/3682883764 >> v1:172.21.15.32:6789/0 conn(0x7fb0500b1f20 legacy=0x7fb0501434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 1 -- 172.21.15.3:0/3682883764 shutdown_connections 2024-03-29T11:20:02.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 1 -- 172.21.15.3:0/3682883764 >> 172.21.15.3:0/3682883764 conn(0x7fb050066a00 msgr2=0x7fb050066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:02.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 1 -- 172.21.15.3:0/3682883764 shutdown_connections 2024-03-29T11:20:02.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.933+0000 7fb05480d640 1 -- 172.21.15.3:0/3682883764 wait complete. 2024-03-29T11:20:02.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.934+0000 7fb05480d640 10 monclient: build_initial_monmap 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.934+0000 7fb05480d640 10 monclient: monmap: 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:02.918 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.934+0000 7fb05480d640 1 librados: starting msgr at 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 librados: starting objecter 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 Processor -- start 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 -- start start 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 librados: setting wanted keys 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 librados: calling monclient init 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient: init 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-a con 0x7fb0500b1f20 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-b con 0x7fb050146400 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 10 monclient(hunting): picked mon.noname-c con 0x7fb05014fd10 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:02.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500b6a20 con 0x7fb0500b1f20 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0500b8750 con 0x7fb050146400 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.935+0000 7fb05480d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb05014f7a0 con 0x7fb05014fd10 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.936+0000 7fb05480d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.936+0000 7fb05480d640 10 monclient(hunting): authenticate will time out at 1130.190011s 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb04ed9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fb05014fd10 0x7fb0501df080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53782/0 (socket says 172.21.15.3:53782) 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb04ed9d640 1 -- 172.21.15.3:0/3856820053 learned_addr learned my addr 172.21.15.3:0/3856820053 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 client.?.objecter ms_handle_connect 0x7fb05014fd10 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278172420 0 0) 0x7fb05014f7a0 con 0x7fb05014fd10 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.937+0000 7fb03effd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:20:02.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb030004eb0 con 0x7fb05014fd10 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62204834 0 0) 0x7fb030004eb0 con 0x7fb05014fd10 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb05014f7a0 con 0x7fb05014fd10 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb048001bc0 con 0x7fb05014fd10 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 10 monclient(hunting): dump: 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:02.921 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb03effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb04dd9b640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.32:6789/0 conn(0x7fb050146400 legacy=0x7fb0501de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb04dd9b640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.32:6789/0 conn(0x7fb050146400 legacy=0x7fb0501de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb04dd9b640 1 --1- 172.21.15.3:0/3856820053 >> v1:172.21.15.32:6789/0 conn(0x7fb050146400 0x7fb0501de970 :-1 s=OPENED pgs=682321 cs=1 l=1).handle_message read tag failed 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.938+0000 7fb04dd9b640 1 --1- 172.21.15.3:0/3856820053 >> v1:172.21.15.32:6789/0 conn(0x7fb050146400 0x7fb0501de970 :-1 s=OPENED pgs=682321 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 client.?.objecter ms_handle_connect 0x7fb050146400 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.32:6789/0 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2250739797 0 0) 0x7fb05014f7a0 con 0x7fb05014fd10 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.32:6789/0 conn(0x7fb050146400 legacy=0x7fb0501de970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:20:02.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.3:6789/0 conn(0x7fb0500b1f20 legacy=0x7fb05013cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 1 monclient: found mon.c 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05014f930 con 0x7fb05014fd10 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb03effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 10 monclient: _renew_subs 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0500b79f0 con 0x7fb05014fd10 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 10 monclient: _renew_subs 2024-03-29T11:20:02.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:02.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0500b6fb0 con 0x7fb05014fd10 2024-03-29T11:20:02.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.939+0000 7fb05480d640 1 librados: init done 2024-03-29T11:20:02.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.941+0000 7fb05480d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:02.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.941+0000 7fb05480d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:02.924 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.941+0000 7fb05480d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.941+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb048001de0 con 0x7fb05014fd10 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.941+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05006a400 con 0x7fb05014fd10 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fb048003bf0 con 0x7fb05014fd10 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 10 monclient: dump: 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:02.926 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fb048003e10 con 0x7fb05014fd10 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb04cd99640 4 set_mon_vals no callback set 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fb048084e30 con 0x7fb05014fd10 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 10 client.4439.objecter ms_dispatch 0x7fb050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:02.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.942+0000 7fb03effd640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.943+0000 7fb04e59c640 10 monclient: get_auth_request con 0x7fb03006dcb0 auth_method 0 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.943+0000 7fb03effd640 10 client.4439.objecter ms_handle_connect 0x7fb03006dcb0 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.950+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb048050680 con 0x7fb05014fd10 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.950+0000 7fb03effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:02.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:02.950+0000 7fb03effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:03.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.260+0000 7fb05480d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.260+0000 7fb05480d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.260+0000 7fb05480d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:03.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.260+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0500b72b0 con 0x7fb05014fd10 2024-03-29T11:20:03.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.261+0000 7fb03effd640 1 -- 172.21.15.3:0/3856820053 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fb048052780 con 0x7fb05014fd10 2024-03-29T11:20:03.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.261+0000 7fb03effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:03.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.261+0000 7fb03effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:03.238 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:03.238 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.264+0000 7fb05480d640 10 librados: watch_flush enter 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.264+0000 7fb05480d640 10 librados: watch_flush exit 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.264+0000 7fb05480d640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.264+0000 7fb05480d640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.3:6816/844689376 conn(0x7fb03006dcb0 legacy=0x7fb030070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 10 monclient: shutdown 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 >> v1:172.21.15.3:6790/0 conn(0x7fb05014fd10 legacy=0x7fb0501df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 shutdown_connections 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 >> 172.21.15.3:0/3856820053 conn(0x7fb050066a00 msgr2=0x7fb0501423c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 shutdown_connections 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 -- 172.21.15.3:0/3856820053 wait complete. 2024-03-29T11:20:03.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:03.265+0000 7fb05480d640 1 librados: shutdown 2024-03-29T11:20:03.256 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:10.064 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:10.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.233+0000 7f090d3f3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:10.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.233+0000 7f090d3f3640 10 monclient: build_initial_monmap 2024-03-29T11:20:10.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.233+0000 7f090d3f3640 10 monclient: monmap: 2024-03-29T11:20:10.211 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:10.211 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.211 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:10.234515+0000 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:10.234515+0000 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:10.212 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:10.213 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.235+0000 7f090d3f3640 1 Processor -- start 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.235+0000 7f090d3f3640 1 -- start start 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.235+0000 7f090d3f3640 10 monclient: init 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-b con 0x7f09000389e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-a con 0x7f0900038dc0 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-c con 0x7f0900030660 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09000c54c0 con 0x7f0900038dc0 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0900036430 con 0x7f0900030660 2024-03-29T11:20:10.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0900039840 con 0x7f09000389e0 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090d3f3640 10 monclient(hunting): authenticate will time out at 1137.490486s 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090cbf2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0900030660 0x7f09000d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:34914/0 (socket says 172.21.15.3:34914) 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.236+0000 7f090cbf2640 1 -- 172.21.15.3:0/1359304943 learned_addr learned my addr 172.21.15.3:0/1359304943 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893673980 0 0) 0x7f0900039840 con 0x7f09000389e0 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8003520 con 0x7f09000389e0 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040700675 0 0) 0x7f0900036430 con 0x7f0900030660 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0900039840 con 0x7f0900030660 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256779016 0 0) 0x7f09000c54c0 con 0x7f0900038dc0 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0900036430 con 0x7f0900038dc0 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.237+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117549186 0 0) 0x7f0900039840 con 0x7f0900030660 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 >> v1:172.21.15.3:6790/0 conn(0x7f09000389e0 legacy=0x7f0900033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 >> v1:172.21.15.3:6789/0 conn(0x7f0900038dc0 legacy=0x7f09000cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:10.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0900035690 con 0x7f0900030660 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f08fc001610 con 0x7f0900030660 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: dump: 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.218 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f090d3f3640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f090d3f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08fc001770 con 0x7f0900030660 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f090d3f3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f090d3f3640 4 set_mon_vals no callback set 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 1 -- 172.21.15.3:0/1359304943 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f08fc001b80 con 0x7f0900030660 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.238+0000 7f0906ffd640 10 monclient: dump: 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:10.220 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 10 monclient: shutdown 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 1 -- 172.21.15.3:0/1359304943 >> v1:172.21.15.32:6789/0 conn(0x7f0900030660 legacy=0x7f09000d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 1 -- 172.21.15.3:0/1359304943 shutdown_connections 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 1 -- 172.21.15.3:0/1359304943 >> 172.21.15.3:0/1359304943 conn(0x7f0900005c00 msgr2=0x7f0900006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.239+0000 7f090d3f3640 1 -- 172.21.15.3:0/1359304943 shutdown_connections 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.240+0000 7f090d3f3640 1 -- 172.21.15.3:0/1359304943 wait complete. 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.241+0000 7f090d3f3640 10 monclient: build_initial_monmap 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.241+0000 7f090d3f3640 10 monclient: monmap: 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:10.225 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.241+0000 7f090d3f3640 1 librados: starting msgr at 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.241+0000 7f090d3f3640 1 librados: starting objecter 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.242+0000 7f090d3f3640 1 Processor -- start 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.242+0000 7f090d3f3640 1 -- start start 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.242+0000 7f090d3f3640 1 librados: setting wanted keys 2024-03-29T11:20:10.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.242+0000 7f090d3f3640 1 librados: calling monclient init 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.242+0000 7f090d3f3640 10 monclient: init 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-c con 0x7f0900030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-b con 0x7f09000389e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f0907fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f0900030660 0x7f0900169a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:37776/0 (socket says 172.21.15.3:37776) 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f0907fff640 1 -- 172.21.15.3:0/1275491663 learned_addr learned my addr 172.21.15.3:0/1275491663 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient(hunting): picked mon.noname-a con 0x7f09000c5650 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0900169570 con 0x7f09000c5650 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f090000acb0 con 0x7f09000389e0 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09000c7ab0 con 0x7f0900030660 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f090d3f3640 10 monclient(hunting): authenticate will time out at 1137.497830s 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f0904ff9640 10 client.?.objecter ms_handle_connect 0x7f0900030660 2024-03-29T11:20:10.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.243+0000 7f0904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 client.?.objecter ms_handle_connect 0x7f09000389e0 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392054247 0 0) 0x7f09000c7ab0 con 0x7f0900030660 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08ec004eb0 con 0x7f0900030660 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929233887 0 0) 0x7f090000acb0 con 0x7f09000389e0 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.244+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09000c7ab0 con 0x7f09000389e0 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 client.?.objecter ms_handle_connect 0x7f09000c5650 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1285850852 0 0) 0x7f08ec004eb0 con 0x7f0900030660 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f090000acb0 con 0x7f0900030660 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782052727 0 0) 0x7f09000c7ab0 con 0x7f09000389e0 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08ec004eb0 con 0x7f09000389e0 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f090804fdd0 con 0x7f0900030660 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.229 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f08f40017f0 con 0x7f09000389e0 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.245+0000 7f0904ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:10.230 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640648331 0 0) 0x7f0900169570 con 0x7f09000c5650 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:20:10.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09000c7ab0 con 0x7f09000c5650 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1857409859 0 0) 0x7f08ec004eb0 con 0x7f09000389e0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 >> v1:172.21.15.3:6790/0 conn(0x7f0900030660 legacy=0x7f0900169a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 >> v1:172.21.15.3:6789/0 conn(0x7f09000c5650 legacy=0x7f0900174ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 monclient: found mon.b 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090016a110 con 0x7f09000389e0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.246+0000 7f0904ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08f4001950 con 0x7f09000389e0 2024-03-29T11:20:10.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f08f40037b0 con 0x7f09000389e0 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f0904ff9640 10 monclient: dump: 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:10.233 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.247+0000 7f09067fc640 4 set_mon_vals no callback set 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 10 monclient: _renew_subs 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0900009a40 con 0x7f09000389e0 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 10 monclient: _renew_subs 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0900166c10 con 0x7f09000389e0 2024-03-29T11:20:10.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.248+0000 7f090d3f3640 1 librados: init done 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.250+0000 7f090d3f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.250+0000 7f090d3f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.250+0000 7f090d3f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.250+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0900007b90 con 0x7f09000389e0 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.251+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f08f4019780 con 0x7f09000389e0 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f08f4083790 con 0x7f09000389e0 2024-03-29T11:20:10.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0904ff9640 10 client.4435.objecter ms_dispatch 0x7f0908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0904ff9640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0904ff9640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0904ff9640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.252+0000 7f0907fff640 10 monclient: get_auth_request con 0x7f08ec06dd70 auth_method 0 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.253+0000 7f0904ff9640 10 client.4435.objecter ms_handle_connect 0x7f08ec06dd70 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.258+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08f404eea0 con 0x7f09000389e0 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.258+0000 7f0904ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:10.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.258+0000 7f0904ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:10.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.668+0000 7f090d3f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.668+0000 7f090d3f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.668+0000 7f090d3f3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:10.645 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.668+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f090016a2a0 con 0x7f09000389e0 2024-03-29T11:20:10.646 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.669+0000 7f0904ff9640 1 -- 172.21.15.3:0/1275491663 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f08f4050fa0 con 0x7f09000389e0 2024-03-29T11:20:10.646 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.670+0000 7f0904ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:10.646 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.670+0000 7f0904ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:10.646 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:10.647 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.673+0000 7f090d3f3640 10 librados: watch_flush enter 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 10 librados: watch_flush exit 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 10 client.4435.objecter successfully canceled tick 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 >> v1:172.21.15.3:6816/844689376 conn(0x7f08ec06dd70 legacy=0x7f08ec070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 10 monclient: shutdown 2024-03-29T11:20:10.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 >> v1:172.21.15.32:6789/0 conn(0x7f09000389e0 legacy=0x7f0900166500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 shutdown_connections 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 >> 172.21.15.3:0/1275491663 conn(0x7f0900005c00 msgr2=0x7f09000324e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 shutdown_connections 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 -- 172.21.15.3:0/1275491663 wait complete. 2024-03-29T11:20:10.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:10.674+0000 7f090d3f3640 1 librados: shutdown 2024-03-29T11:20:10.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:17.474 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.632+0000 7f52ec40f640 10 monclient: get_monmap_and_config 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.632+0000 7f52ec40f640 10 monclient: build_initial_monmap 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.632+0000 7f52ec40f640 10 monclient: monmap: 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:17.633190+0000 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:17.633190+0000 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:17.609 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:17.610 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:17.610 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.633+0000 7f52ec40f640 1 Processor -- start 2024-03-29T11:20:17.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.633+0000 7f52ec40f640 1 -- start start 2024-03-29T11:20:17.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.633+0000 7f52ec40f640 10 monclient: init 2024-03-29T11:20:17.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:17.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:17.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-c con 0x7f52e40b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:17.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-b con 0x7f52e40b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:17.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-a con 0x7f52e40b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40aab60 con 0x7f52e40b3560 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.634+0000 7f52ec40f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40aae40 con 0x7f52e40b28d0 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52ec40f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e40ab120 con 0x7f52e40b2cb0 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52ec40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e99aa640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f52e40b2cb0 0x7f52e4142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47390/0 (socket says 172.21.15.3:47390) 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52ec40f640 10 monclient(hunting): authenticate will time out at 1144.889123s 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e99aa640 1 -- 172.21.15.3:0/3051440705 learned_addr learned my addr 172.21.15.3:0/3051440705 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44064543 0 0) 0x7f52e40ab120 con 0x7f52e40b2cb0 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:20:17.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52d0003520 con 0x7f52e40b2cb0 2024-03-29T11:20:17.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953205591 0 0) 0x7f52d0003520 con 0x7f52e40b2cb0 2024-03-29T11:20:17.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 >> v1:172.21.15.32:6789/0 conn(0x7f52e40b28d0 legacy=0x7f52e413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:17.614 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.635+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 >> v1:172.21.15.3:6789/0 conn(0x7f52e40b3560 legacy=0x7f52e41428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.636+0000 7f52e91a9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.636+0000 7f52e91a9640 1 monclient: found mon.noname-b 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.636+0000 7f52e91a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.636+0000 7f52e91a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e413c380 con 0x7f52e40b2cb0 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: _finish_auth 0 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f52dc001a10 con 0x7f52e40b2cb0 2024-03-29T11:20:17.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: dump: 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52ec40f640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:20:17.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52ec40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52dc001be0 con 0x7f52e40b2cb0 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 1 -- 172.21.15.3:0/3051440705 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f52dc002030 con 0x7f52e40b2cb0 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52ec40f640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52ec40f640 4 set_mon_vals no callback set 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.637+0000 7f52e91a9640 10 monclient: dump: 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.618 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.638+0000 7f52ec40f640 10 monclient: shutdown 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.638+0000 7f52ec40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.638+0000 7f52ec40f640 1 -- 172.21.15.3:0/3051440705 >> v1:172.21.15.3:6790/0 conn(0x7f52e40b2cb0 legacy=0x7f52e4142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.638+0000 7f52ec40f640 1 -- 172.21.15.3:0/3051440705 shutdown_connections 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.638+0000 7f52ec40f640 1 -- 172.21.15.3:0/3051440705 >> 172.21.15.3:0/3051440705 conn(0x7f52e4066a00 msgr2=0x7f52e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.639+0000 7f52ec40f640 1 -- 172.21.15.3:0/3051440705 shutdown_connections 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.639+0000 7f52ec40f640 1 -- 172.21.15.3:0/3051440705 wait complete. 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.639+0000 7f52ec40f640 10 monclient: build_initial_monmap 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.639+0000 7f52ec40f640 10 monclient: monmap: 2024-03-29T11:20:17.619 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:17.620 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 librados: starting msgr at 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 librados: starting objecter 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 Processor -- start 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 -- start start 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 librados: setting wanted keys 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 1 librados: calling monclient init 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.640+0000 7f52ec40f640 10 monclient: init 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ec40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ec40f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-c con 0x7f52e40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ea1ab640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f52e40b28d0 0x7f52e413f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48124/0 (socket says 172.21.15.3:48124) 2024-03-29T11:20:17.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ea1ab640 1 -- 172.21.15.3:0/1035280023 learned_addr learned my addr 172.21.15.3:0/1035280023 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.641+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-b con 0x7f52e413c630 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52e40b28d0 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 10 monclient(hunting): picked mon.noname-a con 0x7f52e41dff60 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e413f920 con 0x7f52e41dff60 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e412f090 con 0x7f52e413c630 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e412f370 con 0x7f52e40b28d0 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.642+0000 7f52ec40f640 10 monclient(hunting): authenticate will time out at 1144.896756s 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.644+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52e41dff60 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.644+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.644+0000 7f52daffd640 10 client.?.objecter ms_handle_connect 0x7f52e413c630 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.644+0000 7f52daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:17.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523220963 0 0) 0x7f52e412f090 con 0x7f52e413c630 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52c8004eb0 con 0x7f52e413c630 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485455305 0 0) 0x7f52e413f920 con 0x7f52e41dff60 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e412f090 con 0x7f52e41dff60 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147323520 0 0) 0x7f52e412f370 con 0x7f52e40b28d0 2024-03-29T11:20:17.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52e413f920 con 0x7f52e40b28d0 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1093202857 0 0) 0x7f52c8004eb0 con 0x7f52e413c630 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52e412f370 con 0x7f52e413c630 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f52dc001ef0 con 0x7f52e413c630 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.645+0000 7f52daffd640 10 monclient(hunting): dump: 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.627 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2631536807 0 0) 0x7f52e412f090 con 0x7f52e41dff60 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52c8004eb0 con 0x7f52e41dff60 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f52e00016f0 con 0x7f52e41dff60 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient(hunting): dump: 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:17.628 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2954527488 0 0) 0x7f52c8004eb0 con 0x7f52e41dff60 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 >> v1:172.21.15.3:6790/0 conn(0x7f52e40b28d0 legacy=0x7f52e413f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 >> v1:172.21.15.32:6789/0 conn(0x7f52e413c630 legacy=0x7f52e413cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 monclient: found mon.a 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:17.629 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e412e2a0 con 0x7f52e41dff60 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 10 monclient: _renew_subs 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e413eca0 con 0x7f52e41dff60 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 10 monclient: _renew_subs 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e41e3a70 con 0x7f52e41dff60 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.646+0000 7f52ec40f640 1 librados: init done 2024-03-29T11:20:17.630 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52e0001850 con 0x7f52e41dff60 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52e89a8640 4 set_mon_vals no callback set 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f52e00035f0 con 0x7f52e41dff60 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.647+0000 7f52daffd640 10 monclient: dump: 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:17.631 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:17.632 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:17.632 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:17.632 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.648+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f52e0019640 con 0x7f52e41dff60 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f52e0083630 con 0x7f52e41dff60 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52daffd640 10 client.4425.objecter ms_dispatch 0x7f52e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52daffd640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52daffd640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52daffd640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.649+0000 7f52ea1ab640 10 monclient: get_auth_request con 0x7f52c806dd50 auth_method 0 2024-03-29T11:20:17.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.650+0000 7f52daffd640 10 client.4425.objecter ms_handle_connect 0x7f52c806dd50 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.650+0000 7f52ec40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.650+0000 7f52ec40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.650+0000 7f52ec40f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.650+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52cc005080 con 0x7f52e41dff60 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.658+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f52e004ed40 con 0x7f52e41dff60 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.658+0000 7f52daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:17.634 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.658+0000 7f52daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:17.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.976+0000 7f52ec40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:17.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.976+0000 7f52ec40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:17.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.976+0000 7f52ec40f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:17.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.976+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52cc005330 con 0x7f52e41dff60 2024-03-29T11:20:17.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.977+0000 7f52daffd640 1 -- 172.21.15.3:0/1035280023 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f52e0050e40 con 0x7f52e41dff60 2024-03-29T11:20:17.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.977+0000 7f52daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:17.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.977+0000 7f52daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:17.955 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:17.955 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 10 librados: watch_flush enter 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 10 librados: watch_flush exit 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 >> v1:172.21.15.3:6816/844689376 conn(0x7f52c806dd50 legacy=0x7f52c8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 10 monclient: shutdown 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 >> v1:172.21.15.3:6789/0 conn(0x7f52e41dff60 legacy=0x7f52e41e2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 shutdown_connections 2024-03-29T11:20:17.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 >> 172.21.15.3:0/1035280023 conn(0x7f52e4066a00 msgr2=0x7f52e413fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:17.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 shutdown_connections 2024-03-29T11:20:17.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 -- 172.21.15.3:0/1035280023 wait complete. 2024-03-29T11:20:17.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:17.981+0000 7f52ec40f640 1 librados: shutdown 2024-03-29T11:20:17.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:24.779 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:24.930 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.953+0000 7f091759e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.953+0000 7f091759e640 10 monclient: build_initial_monmap 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.953+0000 7f091759e640 10 monclient: monmap: 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:24.955101+0000 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:24.955101+0000 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:24.931 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:24.932 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.954+0000 7f091759e640 1 Processor -- start 2024-03-29T11:20:24.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.955+0000 7f091759e640 1 -- start start 2024-03-29T11:20:24.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.955+0000 7f091759e640 10 monclient: init 2024-03-29T11:20:24.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient(hunting): picked mon.noname-b con 0x7f0910006e30 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient(hunting): picked mon.noname-a con 0x7f0910035e00 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient(hunting): picked mon.noname-c con 0x7f09100c9670 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0910035970 con 0x7f0910035e00 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0910036280 con 0x7f09100c9670 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0910035640 con 0x7f0910006e30 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:24.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091759e640 10 monclient(hunting): authenticate will time out at 1152.211323s 2024-03-29T11:20:24.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091659c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f0910006e30 0x7f09100cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:38324/0 (socket says 172.21.15.3:38324) 2024-03-29T11:20:24.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.956+0000 7f091659c640 1 -- 172.21.15.3:0/174352062 learned_addr learned my addr 172.21.15.3:0/174352062 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:24.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918445164 0 0) 0x7f0910035640 con 0x7f0910006e30 2024-03-29T11:20:24.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:24.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:24.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:24.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.958+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0900003520 con 0x7f0910006e30 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 573200405 0 0) 0x7f0900003520 con 0x7f0910006e30 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 >> v1:172.21.15.32:6789/0 conn(0x7f09100c9670 legacy=0x7f09100d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 >> v1:172.21.15.3:6789/0 conn(0x7f0910035e00 legacy=0x7f09100c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 1 monclient: found mon.noname-b 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0910034e20 con 0x7f0910006e30 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 10 monclient: _finish_auth 0 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.960+0000 7f091559a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091759e640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:20:24.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f091804f730 con 0x7f0910006e30 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091559a640 10 monclient: dump: 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:24.940 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.961+0000 7f091759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09180008d0 con 0x7f0910006e30 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 1 -- 172.21.15.3:0/174352062 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f091804f080 con 0x7f0910006e30 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091559a640 10 monclient: dump: 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:24.941 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:24.942 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.963+0000 7f091759e640 4 set_mon_vals no callback set 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.964+0000 7f091759e640 10 monclient: shutdown 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.964+0000 7f091759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.964+0000 7f091759e640 1 -- 172.21.15.3:0/174352062 >> v1:172.21.15.3:6790/0 conn(0x7f0910006e30 legacy=0x7f09100cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.964+0000 7f091759e640 1 -- 172.21.15.3:0/174352062 shutdown_connections 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.964+0000 7f091759e640 1 -- 172.21.15.3:0/174352062 >> 172.21.15.3:0/174352062 conn(0x7f0910005c00 msgr2=0x7f0910006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.965+0000 7f091759e640 1 -- 172.21.15.3:0/174352062 shutdown_connections 2024-03-29T11:20:24.943 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.965+0000 7f091759e640 1 -- 172.21.15.3:0/174352062 wait complete. 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.966+0000 7f091759e640 10 monclient: build_initial_monmap 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.966+0000 7f091759e640 10 monclient: monmap: 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:24.944 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.966+0000 7f091759e640 1 librados: starting msgr at 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.966+0000 7f091759e640 1 librados: starting objecter 2024-03-29T11:20:24.945 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.966+0000 7f091759e640 1 Processor -- start 2024-03-29T11:20:24.946 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 -- start start 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 librados: setting wanted keys 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 librados: calling monclient init 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient: init 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient(hunting): picked mon.noname-c con 0x7f0910006e30 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient(hunting): picked mon.noname-b con 0x7f0910035e00 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient(hunting): picked mon.noname-a con 0x7f09100c9670 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100cdab0 con 0x7f09100c9670 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100c1610 con 0x7f0910035e00 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09100cb4c0 con 0x7f0910006e30 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:24.947 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.967+0000 7f091759e640 10 monclient(hunting): authenticate will time out at 1152.222944s 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f091659c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f0910006e30 0x7f0910033580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40944/0 (socket says 172.21.15.3:40944) 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f091659c640 1 -- 172.21.15.3:0/2884759402 learned_addr learned my addr 172.21.15.3:0/2884759402 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 client.?.objecter ms_handle_connect 0x7f0910006e30 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558039488 0 0) 0x7f09100cb4c0 con 0x7f0910006e30 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:20:24.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.970+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8004eb0 con 0x7f0910006e30 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 client.?.objecter ms_handle_connect 0x7f09100c9670 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 client.?.objecter ms_handle_connect 0x7f0910035e00 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779011108 0 0) 0x7f09100cdab0 con 0x7f09100c9670 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09100cb4c0 con 0x7f09100c9670 2024-03-29T11:20:24.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3986662476 0 0) 0x7f08f8004eb0 con 0x7f0910006e30 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09100cdab0 con 0x7f0910006e30 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299846948 0 0) 0x7f09100c1610 con 0x7f0910035e00 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8004eb0 con 0x7f0910035e00 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f091804f430 con 0x7f0910006e30 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:24.956 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.973+0000 7f0906ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3981628821 0 0) 0x7f09100cdab0 con 0x7f0910006e30 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 >> v1:172.21.15.32:6789/0 conn(0x7f0910035e00 legacy=0x7f09100c07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 >> v1:172.21.15.3:6789/0 conn(0x7f09100c9670 legacy=0x7f09100c0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 1 monclient: found mon.c 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:24.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09100cdd20 con 0x7f0910006e30 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.974+0000 7f0906ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 10 monclient: _renew_subs 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09100ce000 con 0x7f0910006e30 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 10 monclient: _renew_subs 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0910033e70 con 0x7f0910006e30 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.975+0000 7f091759e640 1 librados: init done 2024-03-29T11:20:24.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f091804fdd0 con 0x7f0910006e30 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f091759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f091759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f091759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0910007b90 con 0x7f0910006e30 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0914d99640 4 set_mon_vals no callback set 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0918067be0 con 0x7f0910006e30 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.976+0000 7f0906ffd640 10 monclient: dump: 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:24.959 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:24.960 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:24.960 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:24.960 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:24.960 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:24.960 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f0918067dc0 con 0x7f0910006e30 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f09180cf140 con 0x7f0910006e30 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 10 client.4457.objecter ms_dispatch 0x7f0918000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.978+0000 7f0906ffd640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.979+0000 7f0915d9b640 10 monclient: get_auth_request con 0x7f08f806de80 auth_method 0 2024-03-29T11:20:24.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.979+0000 7f0906ffd640 10 client.4457.objecter ms_handle_connect 0x7f08f806de80 2024-03-29T11:20:24.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.987+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09180cb3e0 con 0x7f0910006e30 2024-03-29T11:20:24.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.987+0000 7f0906ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:24.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:24.987+0000 7f0906ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:25.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.322+0000 7f091759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.322+0000 7f091759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.322+0000 7f091759e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:25.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.322+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0910005120 con 0x7f0910006e30 2024-03-29T11:20:25.302 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:25.302 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:25.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.323+0000 7f0906ffd640 1 -- 172.21.15.3:0/2884759402 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f09180cb580 con 0x7f0910006e30 2024-03-29T11:20:25.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.323+0000 7f0906ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:25.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.323+0000 7f0906ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:25.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.326+0000 7f091759e640 10 librados: watch_flush enter 2024-03-29T11:20:25.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.326+0000 7f091759e640 10 librados: watch_flush exit 2024-03-29T11:20:25.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.326+0000 7f091759e640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-29T11:20:25.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.326+0000 7f091759e640 10 client.4457.objecter successfully canceled tick 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 >> v1:172.21.15.3:6816/844689376 conn(0x7f08f806de80 legacy=0x7f08f8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 10 monclient: shutdown 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 >> v1:172.21.15.3:6790/0 conn(0x7f0910006e30 legacy=0x7f0910033580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 shutdown_connections 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 >> 172.21.15.3:0/2884759402 conn(0x7f0910005c00 msgr2=0x7f091003c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 shutdown_connections 2024-03-29T11:20:25.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.327+0000 7f091759e640 1 -- 172.21.15.3:0/2884759402 wait complete. 2024-03-29T11:20:25.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:25.329+0000 7f091759e640 1 librados: shutdown 2024-03-29T11:20:25.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:32.131 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:32.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.312+0000 7ff8f359e640 10 monclient: get_monmap_and_config 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.312+0000 7ff8f359e640 10 monclient: build_initial_monmap 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.313+0000 7ff8f359e640 10 monclient: monmap: 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:32.313856+0000 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:32.313856+0000 2024-03-29T11:20:32.290 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:32.291 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.291 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:32.291 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:32.291 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:32.291 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.314+0000 7ff8f359e640 1 Processor -- start 2024-03-29T11:20:32.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.314+0000 7ff8f359e640 1 -- start start 2024-03-29T11:20:32.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.314+0000 7ff8f359e640 10 monclient: init 2024-03-29T11:20:32.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.315+0000 7ff8f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:32.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.315+0000 7ff8f359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ec006e30 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec035e00 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ec0c9670 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec035970 con 0x7ff8ec006e30 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec036280 con 0x7ff8ec035e00 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec035640 con 0x7ff8ec0c9670 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:32.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.316+0000 7ff8f359e640 10 monclient(hunting): authenticate will time out at 1159.570871s 2024-03-29T11:20:32.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.319+0000 7ff8f259c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ff8ec006e30 0x7ff8ec0cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52025/0 (socket says 172.21.15.3:52025) 2024-03-29T11:20:32.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.319+0000 7ff8f259c640 1 -- 172.21.15.3:0/1483794753 learned_addr learned my addr 172.21.15.3:0/1483794753 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:32.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.320+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420753924 0 0) 0x7ff8ec035970 con 0x7ff8ec006e30 2024-03-29T11:20:32.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.320+0000 7ff8f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.320+0000 7ff8f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.320+0000 7ff8f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.320+0000 7ff8f159a640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:20:32.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.321+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc003520 con 0x7ff8ec006e30 2024-03-29T11:20:32.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157545778 0 0) 0x7ff8dc003520 con 0x7ff8ec006e30 2024-03-29T11:20:32.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 >> v1:172.21.15.3:6790/0 conn(0x7ff8ec0c9670 legacy=0x7ff8ec0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 >> v1:172.21.15.32:6789/0 conn(0x7ff8ec035e00 legacy=0x7ff8ec0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 monclient: found mon.noname-a 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec034e20 con 0x7ff8ec006e30 2024-03-29T11:20:32.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff8e4001690 con 0x7ff8ec006e30 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.323+0000 7ff8f159a640 10 monclient: dump: 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.303 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e4001860 con 0x7ff8ec006e30 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 1 -- 172.21.15.3:0/1483794753 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff8e4001c70 con 0x7ff8ec006e30 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f159a640 10 monclient: dump: 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:32.304 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f359e640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.324+0000 7ff8f359e640 4 set_mon_vals no callback set 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.325+0000 7ff8f359e640 10 monclient: shutdown 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.325+0000 7ff8f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.325+0000 7ff8f359e640 1 -- 172.21.15.3:0/1483794753 >> v1:172.21.15.3:6789/0 conn(0x7ff8ec006e30 legacy=0x7ff8ec0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.325+0000 7ff8f359e640 1 -- 172.21.15.3:0/1483794753 shutdown_connections 2024-03-29T11:20:32.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.325+0000 7ff8f359e640 1 -- 172.21.15.3:0/1483794753 >> 172.21.15.3:0/1483794753 conn(0x7ff8ec005c00 msgr2=0x7ff8ec006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.328+0000 7ff8f359e640 1 -- 172.21.15.3:0/1483794753 shutdown_connections 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.328+0000 7ff8f359e640 1 -- 172.21.15.3:0/1483794753 wait complete. 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.329+0000 7ff8f359e640 10 monclient: build_initial_monmap 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.329+0000 7ff8f359e640 10 monclient: monmap: 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:32.307 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.329+0000 7ff8f359e640 1 librados: starting msgr at 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 1 librados: starting objecter 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 1 Processor -- start 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 1 -- start start 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 1 librados: setting wanted keys 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 1 librados: calling monclient init 2024-03-29T11:20:32.308 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.330+0000 7ff8f359e640 10 monclient: init 2024-03-29T11:20:32.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.334+0000 7ff8f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:32.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.334+0000 7ff8f359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:32.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.334+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff8ec006e30 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:32.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.336+0000 7ff8f259c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7ff8ec006e30 0x7ff8ec164a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:45786/0 (socket says 172.21.15.3:45786) 2024-03-29T11:20:32.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.336+0000 7ff8f259c640 1 -- 172.21.15.3:0/3460491328 learned_addr learned my addr 172.21.15.3:0/3460491328 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:32.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.336+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff8ec035e00 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.336+0000 7ff8f359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff8ec0c9670 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.336+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec1645a0 con 0x7ff8ec0c9670 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.337+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0c2850 con 0x7ff8ec006e30 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.337+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ec0cf0e0 con 0x7ff8ec035e00 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.337+0000 7ff8f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:32.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.337+0000 7ff8f359e640 10 monclient(hunting): authenticate will time out at 1159.591193s 2024-03-29T11:20:32.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8ec006e30 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027840683 0 0) 0x7ff8ec0c2850 con 0x7ff8ec006e30 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.339+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d4004eb0 con 0x7ff8ec006e30 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 client.?.objecter ms_handle_connect 0x7ff8ec035e00 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549508495 0 0) 0x7ff8ec0cf0e0 con 0x7ff8ec035e00 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:32.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8ec0c2850 con 0x7ff8ec035e00 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2701955935 0 0) 0x7ff8ec0c2850 con 0x7ff8ec035e00 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8ec0cf0e0 con 0x7ff8ec035e00 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff8e8001890 con 0x7ff8ec035e00 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.321 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.340+0000 7ff8e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1394320767 0 0) 0x7ff8ec0cf0e0 con 0x7ff8ec035e00 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 >> v1:172.21.15.32:6789/0 conn(0x7ff8ec006e30 legacy=0x7ff8ec164a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 >> v1:172.21.15.3:6789/0 conn(0x7ff8ec0c9670 legacy=0x7ff8ec0c2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 1 monclient: found mon.c 2024-03-29T11:20:32.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ec0cf3c0 con 0x7ff8ec035e00 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 10 monclient: _renew_subs 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ec033e10 con 0x7ff8ec035e00 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 10 monclient: _renew_subs 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.341+0000 7ff8f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.342+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ec165140 con 0x7ff8ec035e00 2024-03-29T11:20:32.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.342+0000 7ff8f359e640 1 librados: init done 2024-03-29T11:20:32.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8e8001d40 con 0x7ff8ec035e00 2024-03-29T11:20:32.325 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ff8e80037f0 con 0x7ff8ec035e00 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 10 monclient: dump: 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:32.326 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ff8e8003a10 con 0x7ff8ec035e00 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8f0d99640 4 set_mon_vals no callback set 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.344+0000 7ff8f359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7ff8e80838a0 con 0x7ff8ec035e00 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8f359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ec007b90 con 0x7ff8ec035e00 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8f259c640 10 monclient: get_auth_request con 0x7ff8d406ddc0 auth_method 0 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.345+0000 7ff8e2ffd640 10 client.4463.objecter ms_dispatch 0x7ff8f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.346+0000 7ff8e2ffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.346+0000 7ff8e2ffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:32.327 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.346+0000 7ff8e2ffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:20:32.328 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.347+0000 7ff8e2ffd640 10 client.4463.objecter ms_handle_connect 0x7ff8d406ddc0 2024-03-29T11:20:32.333 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.355+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8e804f200 con 0x7ff8ec035e00 2024-03-29T11:20:32.333 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.355+0000 7ff8e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:32.333 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.355+0000 7ff8e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:32.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.708+0000 7ff8f359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.708+0000 7ff8f359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.708+0000 7ff8f359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:20:32.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.708+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8ec164750 con 0x7ff8ec035e00 2024-03-29T11:20:32.687 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:32.687 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:32.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.709+0000 7ff8e2ffd640 1 -- 172.21.15.3:0/3460491328 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7ff8ec164750 con 0x7ff8ec035e00 2024-03-29T11:20:32.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.709+0000 7ff8e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:32.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.709+0000 7ff8e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 10 librados: watch_flush enter 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 10 librados: watch_flush exit 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 >> v1:172.21.15.3:6816/844689376 conn(0x7ff8d406ddc0 legacy=0x7ff8d4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 10 monclient: shutdown 2024-03-29T11:20:32.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 >> v1:172.21.15.3:6790/0 conn(0x7ff8ec035e00 legacy=0x7ff8ec0c1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 shutdown_connections 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 >> 172.21.15.3:0/3460491328 conn(0x7ff8ec005c00 msgr2=0x7ff8ec0c8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 shutdown_connections 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.714+0000 7ff8f359e640 1 -- 172.21.15.3:0/3460491328 wait complete. 2024-03-29T11:20:32.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:32.716+0000 7ff8f359e640 1 librados: shutdown 2024-03-29T11:20:32.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:39.519 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.693+0000 7f6c0ad67640 10 monclient: get_monmap_and_config 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.693+0000 7f6c0ad67640 10 monclient: build_initial_monmap 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.693+0000 7f6c0ad67640 10 monclient: monmap: 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:39.694567+0000 2024-03-29T11:20:39.670 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:39.694567+0000 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:39.671 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.672 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.696+0000 7f6c0ad67640 1 Processor -- start 2024-03-29T11:20:39.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.697+0000 7f6c0ad67640 1 -- start start 2024-03-29T11:20:39.673 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.697+0000 7f6c0ad67640 10 monclient: init 2024-03-29T11:20:39.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:39.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-a con 0x7f6c040a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-c con 0x7f6c040aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-b con 0x7f6c04146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040aed20 con 0x7f6c040a98a0 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040aa8b0 con 0x7f6c040aab00 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c04146a60 con 0x7f6c04146f80 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:39.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.698+0000 7f6c0ad67640 10 monclient(hunting): authenticate will time out at 1166.952465s 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.702+0000 7f6c09304640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6c04146f80 0x7f6c04151cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54649/0 (socket says 172.21.15.3:54649) 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.702+0000 7f6c08b03640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6c040a98a0 0x7f6c040ae610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49263/0 (socket says 172.21.15.3:49263) 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.702+0000 7f6c09304640 1 -- 172.21.15.3:0/2631954983 learned_addr learned my addr 172.21.15.3:0/2631954983 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 653858934 0 0) 0x7f6c040aa8b0 con 0x7f6c040aab00 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:20:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.703+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bec003520 con 0x7f6c040aab00 2024-03-29T11:20:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065727929 0 0) 0x7f6bec003520 con 0x7f6c040aab00 2024-03-29T11:20:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 >> v1:172.21.15.3:6790/0 conn(0x7f6c04146f80 legacy=0x7f6c04151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.680 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 >> v1:172.21.15.3:6789/0 conn(0x7f6c040a98a0 legacy=0x7f6c040ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c04143ef0 con 0x7f6c040aab00 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6bf8001690 con 0x7f6c040aab00 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: dump: 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:39.681 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bf80017f0 con 0x7f6c040aab00 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 1 -- 172.21.15.3:0/2631954983 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6bf8001c40 con 0x7f6c040aab00 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.704+0000 7f6c037fe640 10 monclient: dump: 2024-03-29T11:20:39.682 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:39.683 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:20:39.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:39.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 4 set_mon_vals no callback set 2024-03-29T11:20:39.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 10 monclient: shutdown 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.705+0000 7f6c0ad67640 1 -- 172.21.15.3:0/2631954983 >> v1:172.21.15.32:6789/0 conn(0x7f6c040aab00 legacy=0x7f6c041437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.706+0000 7f6c0ad67640 1 -- 172.21.15.3:0/2631954983 shutdown_connections 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.706+0000 7f6c0ad67640 1 -- 172.21.15.3:0/2631954983 >> 172.21.15.3:0/2631954983 conn(0x7f6c040b3c10 msgr2=0x7f6c040b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.706+0000 7f6c0ad67640 1 -- 172.21.15.3:0/2631954983 shutdown_connections 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.706+0000 7f6c0ad67640 1 -- 172.21.15.3:0/2631954983 wait complete. 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 10 monclient: build_initial_monmap 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 10 monclient: monmap: 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.685 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:39.686 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:39.686 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:39.686 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 librados: starting msgr at 2024-03-29T11:20:39.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 librados: starting objecter 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 Processor -- start 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 -- start start 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 librados: setting wanted keys 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 1 librados: calling monclient init 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.707+0000 7f6c0ad67640 10 monclient: init 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.708+0000 7f6c0ad67640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.708+0000 7f6c0ad67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.708+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-c con 0x7f6c040a98a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-a con 0x7f6c040aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 10 monclient(hunting): picked mon.noname-b con 0x7f6c04146f80 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0406edb0 con 0x7f6c040aab00 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c04142520 con 0x7f6c04146f80 2024-03-29T11:20:39.691 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0406dca0 con 0x7f6c040a98a0 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.712+0000 7f6c0ad67640 10 monclient(hunting): authenticate will time out at 1166.966176s 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.714+0000 7f6c03fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6c040aab00 0x7f6c041deec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49363/0 (socket says 172.21.15.3:49363) 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.714+0000 7f6c03fff640 1 -- 172.21.15.3:0/1810013134 learned_addr learned my addr 172.21.15.3:0/1810013134 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.714+0000 7f6c017fa640 10 client.?.objecter ms_handle_connect 0x7f6c040aab00 2024-03-29T11:20:39.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.714+0000 7f6c017fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336397433 0 0) 0x7f6c0406edb0 con 0x7f6c040aab00 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.716+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bf0004eb0 con 0x7f6c040aab00 2024-03-29T11:20:39.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2628185716 0 0) 0x7f6bf0004eb0 con 0x7f6c040aab00 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c0406edb0 con 0x7f6c040aab00 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6bf8001690 con 0x7f6c040aab00 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient(hunting): dump: 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:39.695 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1754934517 0 0) 0x7f6c0406edb0 con 0x7f6c040aab00 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 >> v1:172.21.15.3:6790/0 conn(0x7f6c040a98a0 legacy=0x7f6c04142c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 >> v1:172.21.15.32:6789/0 conn(0x7f6c04146f80 legacy=0x7f6c041e5c50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 monclient: found mon.a 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:39.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.717+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0406de30 con 0x7f6c040aab00 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c017fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c017fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 10 monclient: _renew_subs 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:39.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c041420d0 con 0x7f6c040aab00 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 10 monclient: _renew_subs 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c041df7b0 con 0x7f6c040aab00 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.718+0000 7f6c0ad67640 1 librados: init done 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.719+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bf8003940 con 0x7f6c040aab00 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.719+0000 7f6c017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.719+0000 7f6c02ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c0ad67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6bf8003d50 con 0x7f6c040aab00 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c017fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c017fa640 10 monclient: dump: 2024-03-29T11:20:39.700 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.720+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f6bf8003f70 con 0x7f6c040aab00 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f6bf8084d50 con 0x7f6c040aab00 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c017fa640 10 client.4437.objecter ms_dispatch 0x7f6c04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c017fa640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:39.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c017fa640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c017fa640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c0ad67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c0ad67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.721+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be4005080 con 0x7f6c040aab00 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.722+0000 7f6c08b03640 10 monclient: get_auth_request con 0x7f6bf006dcf0 auth_method 0 2024-03-29T11:20:39.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.722+0000 7f6c017fa640 10 client.4437.objecter ms_handle_connect 0x7f6bf006dcf0 2024-03-29T11:20:39.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.730+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bf80506b0 con 0x7f6c040aab00 2024-03-29T11:20:39.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.730+0000 7f6c017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:39.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:39.730+0000 7f6c017fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:40.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.057+0000 7f6c0ad67640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.057+0000 7f6c0ad67640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.057+0000 7f6c0ad67640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.057+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6be4005330 con 0x7f6c040aab00 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.059+0000 7f6c017fa640 1 -- 172.21.15.3:0/1810013134 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f6bf80525e0 con 0x7f6c040aab00 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.059+0000 7f6c017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:40.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.059+0000 7f6c017fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:40.037 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:40.037 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:40.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.064+0000 7f6c0ad67640 10 librados: watch_flush enter 2024-03-29T11:20:40.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.064+0000 7f6c0ad67640 10 librados: watch_flush exit 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 >> v1:172.21.15.3:6816/844689376 conn(0x7f6bf006dcf0 legacy=0x7f6bf00701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 10 monclient: shutdown 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:40.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.065+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 >> v1:172.21.15.3:6789/0 conn(0x7f6c040aab00 legacy=0x7f6c041deec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:40.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.067+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 shutdown_connections 2024-03-29T11:20:40.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.067+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 >> 172.21.15.3:0/1810013134 conn(0x7f6c040b3c10 msgr2=0x7f6c040b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:40.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.068+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 shutdown_connections 2024-03-29T11:20:40.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.068+0000 7f6c0ad67640 1 -- 172.21.15.3:0/1810013134 wait complete. 2024-03-29T11:20:40.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:40.068+0000 7f6c0ad67640 1 librados: shutdown 2024-03-29T11:20:40.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:46.871 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.048+0000 7fbd06e72640 10 monclient: get_monmap_and_config 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.048+0000 7fbd06e72640 10 monclient: build_initial_monmap 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.049+0000 7fbd06e72640 10 monclient: monmap: 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:47.049949+0000 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:47.049949+0000 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.026 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:47.027 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:47.027 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:47.027 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.051+0000 7fbd06e72640 1 Processor -- start 2024-03-29T11:20:47.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.052+0000 7fbd06e72640 1 -- start start 2024-03-29T11:20:47.028 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.052+0000 7fbd06e72640 10 monclient: init 2024-03-29T11:20:47.029 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.053+0000 7fbd06e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:47.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.053+0000 7fbd06e72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:47.030 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.053+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-b con 0x7fbd000a94a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:47.031 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-a con 0x7fbd000a9880 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-c con 0x7fbd0013ee00 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd000aadc0 con 0x7fbd000a9880 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd0013bd10 con 0x7fbd0013ee00 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd000ab0a0 con 0x7fbd000a94a0 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:47.032 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.054+0000 7fbd06e72640 10 monclient(hunting): authenticate will time out at 1174.308355s 2024-03-29T11:20:47.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.057+0000 7fbd05e70640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fbd000a94a0 0x7fbd0013b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52143/0 (socket says 172.21.15.3:52143) 2024-03-29T11:20:47.033 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.057+0000 7fbd05e70640 1 -- 172.21.15.3:0/1808129715 learned_addr learned my addr 172.21.15.3:0/1808129715 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.059+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203293040 0 0) 0x7fbd000ab0a0 con 0x7fbd000a94a0 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.059+0000 7fbd04e6e640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.059+0000 7fbd04e6e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.059+0000 7fbd04e6e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.060+0000 7fbd04e6e640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:20:47.036 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.060+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcec003520 con 0x7fbd000a94a0 2024-03-29T11:20:47.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1511812396 0 0) 0x7fbcec003520 con 0x7fbd000a94a0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 >> v1:172.21.15.32:6789/0 conn(0x7fbd0013ee00 legacy=0x7fbd001499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 >> v1:172.21.15.3:6789/0 conn(0x7fbd000a9880 legacy=0x7fbd0013e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 10 monclient: _finish_hunting 0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 1 monclient: found mon.noname-b 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.062+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0013e1d0 con 0x7fbd000a94a0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 10 monclient: _finish_auth 0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 10 monclient: _check_auth_tickets 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbcfc001690 con 0x7fbd000a94a0 2024-03-29T11:20:47.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.063+0000 7fbd04e6e640 10 monclient: dump: 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:47.041 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcfc001920 con 0x7fbd000a94a0 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 1 -- 172.21.15.3:0/1808129715 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbcfc001cb0 con 0x7fbd000a94a0 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd04e6e640 10 monclient: dump: 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:47.043 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd06e72640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd06e72640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.064+0000 7fbd06e72640 4 set_mon_vals no callback set 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.065+0000 7fbd06e72640 10 monclient: shutdown 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.065+0000 7fbd06e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.065+0000 7fbd06e72640 1 -- 172.21.15.3:0/1808129715 >> v1:172.21.15.3:6790/0 conn(0x7fbd000a94a0 legacy=0x7fbd0013b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.065+0000 7fbd06e72640 1 -- 172.21.15.3:0/1808129715 shutdown_connections 2024-03-29T11:20:47.044 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.065+0000 7fbd06e72640 1 -- 172.21.15.3:0/1808129715 >> 172.21.15.3:0/1808129715 conn(0x7fbd000b3820 msgr2=0x7fbd000b3c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:47.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.066+0000 7fbd06e72640 1 -- 172.21.15.3:0/1808129715 shutdown_connections 2024-03-29T11:20:47.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.068+0000 7fbd06e72640 1 -- 172.21.15.3:0/1808129715 wait complete. 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.070+0000 7fbd06e72640 10 monclient: build_initial_monmap 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.070+0000 7fbd06e72640 10 monclient: monmap: 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:47.046 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.047 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:47.047 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:47.047 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:47.047 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.070+0000 7fbd06e72640 1 librados: starting msgr at 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.070+0000 7fbd06e72640 1 librados: starting objecter 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.070+0000 7fbd06e72640 1 Processor -- start 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.071+0000 7fbd06e72640 1 -- start start 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.071+0000 7fbd06e72640 1 librados: setting wanted keys 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.071+0000 7fbd06e72640 1 librados: calling monclient init 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.071+0000 7fbd06e72640 10 monclient: init 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.072+0000 7fbd06e72640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:47.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.072+0000 7fbd06e72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:47.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.072+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-b con 0x7fbd0013ee00 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:47.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.075+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-c con 0x7fbd0013d420 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:47.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.075+0000 7fbd05e70640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbd0013ee00 0x7fbd00071950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:36333/0 (socket says 172.21.15.3:36333) 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06671640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbd0006fb00 0x7fbd001dfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53179/0 (socket says 172.21.15.3:53179) 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06671640 1 -- 172.21.15.3:0/776912260 learned_addr learned my addr 172.21.15.3:0/776912260 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06e72640 10 monclient(hunting): picked mon.noname-a con 0x7fbd0006fb00 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbcf67fc640 10 client.?.objecter ms_handle_connect 0x7fbd0006fb00 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd00072260 con 0x7fbd0006fb00 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbcf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd00070f00 con 0x7fbd0013ee00 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492038947 0 0) 0x7fbd00072260 con 0x7fbd0006fb00 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.079+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd00071170 con 0x7fbd0013d420 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbd06e72640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbd06e72640 10 monclient(hunting): authenticate will time out at 1174.334224s 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce4004eb0 con 0x7fbd0006fb00 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 client.?.objecter ms_handle_connect 0x7fbd0013ee00 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 client.?.objecter ms_handle_connect 0x7fbd0013d420 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.080+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085921784 0 0) 0x7fbce4004eb0 con 0x7fbd0006fb00 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd00072260 con 0x7fbd0006fb00 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbcf8001c30 con 0x7fbd0006fb00 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): dump: 2024-03-29T11:20:47.062 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097084068 0 0) 0x7fbd00071170 con 0x7fbd0013d420 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:47.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce4004eb0 con 0x7fbd0013d420 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295053116 0 0) 0x7fbd00072260 con 0x7fbd0006fb00 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.081+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 >> v1:172.21.15.3:6790/0 conn(0x7fbd0013d420 legacy=0x7fbd0006f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 >> v1:172.21.15.32:6789/0 conn(0x7fbd0013ee00 legacy=0x7fbd00071950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 1 monclient: found mon.a 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd0006ff30 con 0x7fbd0006fb00 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:47.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbcf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbd06e72640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbd06e72640 10 monclient: _renew_subs 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbd06e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd001e1610 con 0x7fbd0006fb00 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.082+0000 7fbd06e72640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbd06e72640 10 monclient: _renew_subs 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbd06e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd001e1980 con 0x7fbd0006fb00 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbd06e72640 1 librados: init done 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbcf8001d90 con 0x7fbd0006fb00 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbcf8003b30 con 0x7fbd0006fb00 2024-03-29T11:20:47.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf67fc640 10 monclient: dump: 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:47.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.083+0000 7fbcf77fe640 4 set_mon_vals no callback set 2024-03-29T11:20:47.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.084+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fbcf8003c90 con 0x7fbd0006fb00 2024-03-29T11:20:47.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.085+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fbcf8084ce0 con 0x7fbd0006fb00 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.085+0000 7fbcf67fc640 10 client.4443.objecter ms_dispatch 0x7fbd00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.085+0000 7fbcf67fc640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.085+0000 7fbcf67fc640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.085+0000 7fbcf67fc640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbd06e72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbd06e72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbd06e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd0006a400 con 0x7fbd0006fb00 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbd05e70640 10 monclient: get_auth_request con 0x7fbce406de20 auth_method 0 2024-03-29T11:20:47.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.086+0000 7fbcf67fc640 10 client.4443.objecter ms_handle_connect 0x7fbce406de20 2024-03-29T11:20:47.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.093+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbcf80503f0 con 0x7fbd0006fb00 2024-03-29T11:20:47.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.093+0000 7fbcf67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:47.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.093+0000 7fbcf67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:47.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.421+0000 7fbd06e72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.421+0000 7fbd06e72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.421+0000 7fbd06e72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:20:47.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.421+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd000b2ed0 con 0x7fbd0006fb00 2024-03-29T11:20:47.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.424+0000 7fbcf67fc640 1 -- 172.21.15.3:0/776912260 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fbcf80524f0 con 0x7fbd0006fb00 2024-03-29T11:20:47.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.424+0000 7fbcf67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:47.401 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.424+0000 7fbcf67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:47.402 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:47.402 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:47.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 10 librados: watch_flush enter 2024-03-29T11:20:47.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 10 librados: watch_flush exit 2024-03-29T11:20:47.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T11:20:47.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 10 client.4443.objecter successfully canceled tick 2024-03-29T11:20:47.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 >> v1:172.21.15.3:6816/844689376 conn(0x7fbce406de20 legacy=0x7fbce40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 10 monclient: shutdown 2024-03-29T11:20:47.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:47.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.430+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 >> v1:172.21.15.3:6789/0 conn(0x7fbd0006fb00 legacy=0x7fbd001dfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:47.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.431+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 shutdown_connections 2024-03-29T11:20:47.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.431+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 >> 172.21.15.3:0/776912260 conn(0x7fbd000b3820 msgr2=0x7fbd000b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:47.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.433+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 shutdown_connections 2024-03-29T11:20:47.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.433+0000 7fbd06e72640 1 -- 172.21.15.3:0/776912260 wait complete. 2024-03-29T11:20:47.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:47.436+0000 7fbd06e72640 1 librados: shutdown 2024-03-29T11:20:47.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:54.238 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:54.399 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.422+0000 7f12909e8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.422+0000 7f12909e8640 10 monclient: build_initial_monmap 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.422+0000 7f12909e8640 10 monclient: monmap: 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:20:54.423802+0000 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:20:54.423802+0000 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:54.400 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:20:54.401 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:20:54.401 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.426+0000 7f12909e8640 1 Processor -- start 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.426+0000 7f12909e8640 1 -- start start 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.426+0000 7f12909e8640 10 monclient: init 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1284006870 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:54.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-c con 0x7f1284006c50 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-b con 0x7f12840c78e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1284033a20 con 0x7f1284006870 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12840cb4f0 con 0x7f1284006c50 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1284033c70 con 0x7f12840c78e0 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:54.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.427+0000 7f12909e8640 10 monclient(hunting): authenticate will time out at 1181.681461s 2024-03-29T11:20:54.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.431+0000 7f128affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1284006c50 0x7f12840cade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:36871/0 (socket says 172.21.15.3:36871) 2024-03-29T11:20:54.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.431+0000 7f128affd640 1 -- 172.21.15.3:0/2595439505 learned_addr learned my addr 172.21.15.3:0/2595439505 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:54.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337398520 0 0) 0x7f12840cb4f0 con 0x7f1284006c50 2024-03-29T11:20:54.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:20:54.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.432+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f127c003520 con 0x7f1284006c50 2024-03-29T11:20:54.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.433+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796406554 0 0) 0x7f1284033a20 con 0x7f1284006870 2024-03-29T11:20:54.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.433+0000 7f128a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.433+0000 7f128a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.433+0000 7f128a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.433+0000 7f128a7fc640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12840cb4f0 con 0x7f1284006870 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2682287748 0 0) 0x7f127c003520 con 0x7f1284006c50 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 >> v1:172.21.15.3:6790/0 conn(0x7f12840c78e0 legacy=0x7f12840d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 >> v1:172.21.15.3:6789/0 conn(0x7f1284006870 legacy=0x7f12840c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840c7f00 con 0x7f1284006c50 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:54.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1280001690 con 0x7f1284006c50 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: dump: 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:54.414 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1280001860 con 0x7f1284006c50 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 1 -- 172.21.15.3:0/2595439505 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1280001cb0 con 0x7f1284006c50 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f128a7fc640 10 monclient: dump: 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:54.415 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f12909e8640 5 monclient: authenticate success, global_id 4462 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f12909e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.434+0000 7f12909e8640 4 set_mon_vals no callback set 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.435+0000 7f12909e8640 10 monclient: shutdown 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.435+0000 7f12909e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:54.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.435+0000 7f12909e8640 1 -- 172.21.15.3:0/2595439505 >> v1:172.21.15.32:6789/0 conn(0x7f1284006c50 legacy=0x7f12840cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.439+0000 7f12909e8640 1 -- 172.21.15.3:0/2595439505 shutdown_connections 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.439+0000 7f12909e8640 1 -- 172.21.15.3:0/2595439505 >> 172.21.15.3:0/2595439505 conn(0x7f12840058f0 msgr2=0x7f1284005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.439+0000 7f12909e8640 1 -- 172.21.15.3:0/2595439505 shutdown_connections 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.439+0000 7f12909e8640 1 -- 172.21.15.3:0/2595439505 wait complete. 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.440+0000 7f12909e8640 10 monclient: build_initial_monmap 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.440+0000 7f12909e8640 10 monclient: monmap: 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:20:54.417 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.418 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.440+0000 7f12909e8640 1 librados: starting msgr at 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.440+0000 7f12909e8640 1 librados: starting objecter 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 1 Processor -- start 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 1 -- start start 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 1 librados: setting wanted keys 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 1 librados: calling monclient init 2024-03-29T11:20:54.419 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 10 monclient: init 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.441+0000 7f12909e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.442+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-a con 0x7f1284006870 addr v1:172.21.15.3:6789/0 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-b con 0x7f12840c78e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 10 monclient(hunting): picked mon.noname-c con 0x7f128402daf0 addr v1:172.21.15.3:6790/0 2024-03-29T11:20:54.420 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1284034490 con 0x7f1284006870 2024-03-29T11:20:54.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128402fc00 con 0x7f12840c78e0 2024-03-29T11:20:54.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12840364c0 con 0x7f128402daf0 2024-03-29T11:20:54.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:54.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.444+0000 7f12909e8640 10 monclient(hunting): authenticate will time out at 1181.698471s 2024-03-29T11:20:54.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.447+0000 7f128affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12840c78e0 0x7f1284035030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:36937/0 (socket says 172.21.15.3:36937) 2024-03-29T11:20:54.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.447+0000 7f128affd640 1 -- 172.21.15.3:0/1475579106 learned_addr learned my addr 172.21.15.3:0/1475579106 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:20:54.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.448+0000 7f1277fff640 10 client.?.objecter ms_handle_connect 0x7f12840c78e0 2024-03-29T11:20:54.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.448+0000 7f1277fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.448+0000 7f1277fff640 10 client.?.objecter ms_handle_connect 0x7f128402daf0 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.448+0000 7f1277fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730387869 0 0) 0x7f128402fc00 con 0x7f12840c78e0 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:20:54.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1270004eb0 con 0x7f12840c78e0 2024-03-29T11:20:54.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 client.?.objecter ms_handle_connect 0x7f1284006870 2024-03-29T11:20:54.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016762205 0 0) 0x7f12840364c0 con 0x7f128402daf0 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f128402fc00 con 0x7f128402daf0 2024-03-29T11:20:54.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262552700 0 0) 0x7f1270004eb0 con 0x7f12840c78e0 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.449+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12840364c0 con 0x7f12840c78e0 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198858913 0 0) 0x7f1284034490 con 0x7f1284006870 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1270004eb0 con 0x7f1284006870 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1280001b70 con 0x7f12840c78e0 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): dump: 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:54.429 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:54.430 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2140134109 0 0) 0x7f128402fc00 con 0x7f128402daf0 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1284034490 con 0x7f128402daf0 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1278001730 con 0x7f128402daf0 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.450+0000 7f1277fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 10 monclient(hunting): dump: 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.431 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3008750274 0 0) 0x7f12840364c0 con 0x7f12840c78e0 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 >> v1:172.21.15.3:6790/0 conn(0x7f128402daf0 legacy=0x7f1284167460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 >> v1:172.21.15.3:6789/0 conn(0x7f1284006870 legacy=0x7f1284034920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.451+0000 7f1277fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 1 monclient: found mon.b 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128402e700 con 0x7f12840c78e0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f1277fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 10 monclient: _renew_subs 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128402e9b0 con 0x7f12840c78e0 2024-03-29T11:20:54.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 10 monclient: _renew_subs 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1284168d60 con 0x7f12840c78e0 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.452+0000 7f12909e8640 1 librados: init done 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1280004a90 con 0x7f12840c78e0 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1289ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1280001cd0 con 0x7f12840c78e0 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.453+0000 7f1277fff640 10 monclient: dump: 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.438 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.455+0000 7f12909e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.455+0000 7f12909e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.455+0000 7f12909e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.455+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c005080 con 0x7f12840c78e0 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.455+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f128001ac10 con 0x7f12840c78e0 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f12800849e0 con 0x7f12840c78e0 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f1277fff640 10 client.4465.objecter ms_dispatch 0x7f128c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:54.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f128b7fe640 10 monclient: get_auth_request con 0x7f127006dbe0 auth_method 0 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f1277fff640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f1277fff640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.456+0000 7f1277fff640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.457+0000 7f1277fff640 10 client.4465.objecter ms_handle_connect 0x7f127006dbe0 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.463+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12800502c0 con 0x7f12840c78e0 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.463+0000 7f1277fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:54.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.463+0000 7f1277fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:54.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.842+0000 7f12909e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.842+0000 7f12909e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.842+0000 7f12909e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.842+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f126c005330 con 0x7f12840c78e0 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.843+0000 7f1277fff640 1 -- 172.21.15.3:0/1475579106 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f12800521f0 con 0x7f12840c78e0 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.843+0000 7f1277fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:54.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.843+0000 7f1277fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:54.820 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:20:54.820 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:54.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 10 librados: watch_flush enter 2024-03-29T11:20:54.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 10 librados: watch_flush exit 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 >> v1:172.21.15.3:6816/844689376 conn(0x7f127006dbe0 legacy=0x7f12700700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 10 monclient: shutdown 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.847+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 >> v1:172.21.15.32:6789/0 conn(0x7f12840c78e0 legacy=0x7f1284035030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.848+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 shutdown_connections 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.848+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 >> 172.21.15.3:0/1475579106 conn(0x7f12840058f0 msgr2=0x7f1284005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.848+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 shutdown_connections 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.848+0000 7f12909e8640 1 -- 172.21.15.3:0/1475579106 wait complete. 2024-03-29T11:20:54.825 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:20:54.848+0000 7f12909e8640 1 librados: shutdown 2024-03-29T11:20:54.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:01.653 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:01.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.828+0000 7f4a9f138640 10 monclient: get_monmap_and_config 2024-03-29T11:21:01.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.828+0000 7f4a9f138640 10 monclient: build_initial_monmap 2024-03-29T11:21:01.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.828+0000 7f4a9f138640 10 monclient: monmap: 2024-03-29T11:21:01.805 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:01.829031+0000 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:01.829031+0000 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.806 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:01.807 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:01.807 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:01.807 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.807 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.829+0000 7f4a9f138640 1 Processor -- start 2024-03-29T11:21:01.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.832+0000 7f4a9f138640 1 -- start start 2024-03-29T11:21:01.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.832+0000 7f4a9f138640 10 monclient: init 2024-03-29T11:21:01.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:01.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:01.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-b con 0x7f4a980b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:01.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-c con 0x7f4a980b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:01.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-a con 0x7f4a980b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:01.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980aab60 con 0x7f4a980b3560 2024-03-29T11:21:01.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980aae40 con 0x7f4a980b2cb0 2024-03-29T11:21:01.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980ab120 con 0x7f4a980b28d0 2024-03-29T11:21:01.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:01.811 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.833+0000 7f4a9f138640 10 monclient(hunting): authenticate will time out at 1189.087611s 2024-03-29T11:21:01.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a97fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4a980b2cb0 0x7f4a98142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:47233/0 (socket says 172.21.15.3:47233) 2024-03-29T11:21:01.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a97fff640 1 -- 172.21.15.3:0/2339785945 learned_addr learned my addr 172.21.15.3:0/2339785945 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:01.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920424582 0 0) 0x7f4a980aae40 con 0x7f4a980b2cb0 2024-03-29T11:21:01.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a88003520 con 0x7f4a980b2cb0 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271906780 0 0) 0x7f4a88003520 con 0x7f4a980b2cb0 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 >> v1:172.21.15.3:6790/0 conn(0x7f4a980b28d0 legacy=0x7f4a9813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:01.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.837+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 >> v1:172.21.15.3:6789/0 conn(0x7f4a980b3560 legacy=0x7f4a981428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 1 monclient: found mon.noname-c 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9813c380 con 0x7f4a980b2cb0 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4a8c001690 con 0x7f4a980b2cb0 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.838+0000 7f4a977fe640 10 monclient: dump: 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:01.816 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a8c001860 con 0x7f4a980b2cb0 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 1 -- 172.21.15.3:0/2339785945 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4a8c001cb0 con 0x7f4a980b2cb0 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a977fe640 10 monclient: dump: 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.818 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:01.819 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a9f138640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a9f138640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.841+0000 7f4a9f138640 4 set_mon_vals no callback set 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.842+0000 7f4a9f138640 10 monclient: shutdown 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.842+0000 7f4a9f138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.842+0000 7f4a9f138640 1 -- 172.21.15.3:0/2339785945 >> v1:172.21.15.32:6789/0 conn(0x7f4a980b2cb0 legacy=0x7f4a98142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.843+0000 7f4a9f138640 1 -- 172.21.15.3:0/2339785945 shutdown_connections 2024-03-29T11:21:01.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.843+0000 7f4a9f138640 1 -- 172.21.15.3:0/2339785945 >> 172.21.15.3:0/2339785945 conn(0x7f4a98066a00 msgr2=0x7f4a98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:01.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.846+0000 7f4a9f138640 1 -- 172.21.15.3:0/2339785945 shutdown_connections 2024-03-29T11:21:01.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.846+0000 7f4a9f138640 1 -- 172.21.15.3:0/2339785945 wait complete. 2024-03-29T11:21:01.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 10 monclient: build_initial_monmap 2024-03-29T11:21:01.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 10 monclient: monmap: 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 librados: starting msgr at 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 librados: starting objecter 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 Processor -- start 2024-03-29T11:21:01.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 -- start start 2024-03-29T11:21:01.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 librados: setting wanted keys 2024-03-29T11:21:01.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 1 librados: calling monclient init 2024-03-29T11:21:01.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.847+0000 7f4a9f138640 10 monclient: init 2024-03-29T11:21:01.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.848+0000 7f4a9f138640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:01.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.848+0000 7f4a9f138640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:01.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.851+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-a con 0x7f4a980b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:01.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.851+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-c con 0x7f4a981d7220 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:01.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.851+0000 7f4a9f138640 10 monclient(hunting): picked mon.noname-b con 0x7f4a9812f650 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:01.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.852+0000 7f4a9f138640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a981d77d0 con 0x7f4a980b28d0 2024-03-29T11:21:01.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.852+0000 7f4a9f138640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a981311a0 con 0x7f4a9812f650 2024-03-29T11:21:01.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.852+0000 7f4a9f138640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98131fb0 con 0x7f4a981d7220 2024-03-29T11:21:01.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.852+0000 7f4a9f138640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:01.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.852+0000 7f4a9f138640 10 monclient(hunting): authenticate will time out at 1189.106028s 2024-03-29T11:21:01.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a9ced4640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f4a980b28d0 0x7f4a981318a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:59647/0 (socket says 172.21.15.3:59647) 2024-03-29T11:21:01.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a9ced4640 1 -- 172.21.15.3:0/559764770 learned_addr learned my addr 172.21.15.3:0/559764770 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a957fa640 10 client.?.objecter ms_handle_connect 0x7f4a980b28d0 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a957fa640 10 client.?.objecter ms_handle_connect 0x7f4a981d7220 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.854+0000 7f4a957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396008249 0 0) 0x7f4a981d77d0 con 0x7f4a980b28d0 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a7c004eb0 con 0x7f4a980b28d0 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773420963 0 0) 0x7f4a98131fb0 con 0x7f4a981d7220 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:21:01.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a981d77d0 con 0x7f4a981d7220 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 86553909 0 0) 0x7f4a7c004eb0 con 0x7f4a980b28d0 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a98131fb0 con 0x7f4a980b28d0 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4a84001960 con 0x7f4a980b28d0 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): dump: 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.833 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.855+0000 7f4a957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1672285017 0 0) 0x7f4a98131fb0 con 0x7f4a980b28d0 2024-03-29T11:21:01.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 >> v1:172.21.15.3:6790/0 conn(0x7f4a981d7220 legacy=0x7f4a9812ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 >> v1:172.21.15.32:6789/0 conn(0x7f4a9812f650 legacy=0x7f4a981e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 1 monclient: found mon.a 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a981300d0 con 0x7f4a980b28d0 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 10 monclient: _finish_auth 0 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:21:01.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 10 monclient: _renew_subs 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a98130320 con 0x7f4a980b28d0 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 10 monclient: _renew_subs 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9813ec50 con 0x7f4a980b28d0 2024-03-29T11:21:01.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.857+0000 7f4a9f138640 1 librados: init done 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.859+0000 7f4a9f138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.859+0000 7f4a9f138640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.859+0000 7f4a9f138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.859+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9806a400 con 0x7f4a980b28d0 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.860+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a84001ac0 con 0x7f4a980b28d0 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.860+0000 7f4a957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.860+0000 7f4a96ffd640 4 set_mon_vals no callback set 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.861+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4a840038a0 con 0x7f4a980b28d0 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.861+0000 7f4a957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.861+0000 7f4a957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:01.839 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.861+0000 7f4a957fa640 10 monclient: dump: 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:01.841 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.861+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f4a84003ac0 con 0x7f4a980b28d0 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.862+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f4a84084ba0 con 0x7f4a980b28d0 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.862+0000 7f4a957fa640 10 client.4452.objecter ms_dispatch 0x7f4a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.862+0000 7f4a957fa640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.862+0000 7f4a957fa640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:01.842 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.862+0000 7f4a957fa640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T11:21:01.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.867+0000 7f4a97fff640 10 monclient: get_auth_request con 0x7f4a7c06db40 auth_method 0 2024-03-29T11:21:01.845 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.868+0000 7f4a957fa640 10 client.4452.objecter ms_handle_connect 0x7f4a7c06db40 2024-03-29T11:21:01.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.870+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a84050500 con 0x7f4a980b28d0 2024-03-29T11:21:01.848 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.870+0000 7f4a957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:01.849 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:01.870+0000 7f4a957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:02.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.208+0000 7f4a9f138640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.208+0000 7f4a9f138640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.208+0000 7f4a9f138640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:02.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.208+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a980b5f70 con 0x7f4a980b28d0 2024-03-29T11:21:02.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.210+0000 7f4a957fa640 1 -- 172.21.15.3:0/559764770 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f4a84052430 con 0x7f4a980b28d0 2024-03-29T11:21:02.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.210+0000 7f4a957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:02.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.210+0000 7f4a957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:02.187 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:02.187 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:02.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.213+0000 7f4a9f138640 10 librados: watch_flush enter 2024-03-29T11:21:02.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.213+0000 7f4a9f138640 10 librados: watch_flush exit 2024-03-29T11:21:02.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.213+0000 7f4a9f138640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T11:21:02.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.213+0000 7f4a9f138640 10 client.4452.objecter successfully canceled tick 2024-03-29T11:21:02.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.214+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 >> v1:172.21.15.3:6816/844689376 conn(0x7f4a7c06db40 legacy=0x7f4a7c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:02.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.214+0000 7f4a9f138640 10 monclient: shutdown 2024-03-29T11:21:02.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.214+0000 7f4a9f138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:02.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.214+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 >> v1:172.21.15.3:6789/0 conn(0x7f4a980b28d0 legacy=0x7f4a981318a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:02.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.217+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 shutdown_connections 2024-03-29T11:21:02.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.217+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 >> 172.21.15.3:0/559764770 conn(0x7f4a98066a00 msgr2=0x7f4a98141870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:02.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.219+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 shutdown_connections 2024-03-29T11:21:02.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.219+0000 7f4a9f138640 1 -- 172.21.15.3:0/559764770 wait complete. 2024-03-29T11:21:02.196 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:02.219+0000 7f4a9f138640 1 librados: shutdown 2024-03-29T11:21:02.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:09.020 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:09.171 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.195+0000 7fecaffff640 10 monclient: get_monmap_and_config 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.195+0000 7fecaffff640 10 monclient: build_initial_monmap 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.195+0000 7fecaffff640 10 monclient: monmap: 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:09.196040+0000 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:09.196040+0000 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:09.172 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.196+0000 7fecaffff640 1 Processor -- start 2024-03-29T11:21:09.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.196+0000 7fecaffff640 1 -- start start 2024-03-29T11:21:09.173 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.196+0000 7fecaffff640 10 monclient: init 2024-03-29T11:21:09.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.196+0000 7fecaffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:09.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.196+0000 7fecaffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:09.174 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-b con 0x7feca8006890 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-a con 0x7feca8006c70 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-c con 0x7feca802b4c0 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca802c830 con 0x7feca8006c70 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca802afa0 con 0x7feca802b4c0 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca802aba0 con 0x7feca8006890 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.197+0000 7fecaffff640 10 monclient(hunting): authenticate will time out at 1196.451067s 2024-03-29T11:21:09.175 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.198+0000 7fecaf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feca802b4c0 0x7feca80cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:45192/0 (socket says 172.21.15.3:45192) 2024-03-29T11:21:09.176 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.198+0000 7fecaf7fe640 1 -- 172.21.15.3:0/2790935359 learned_addr learned my addr 172.21.15.3:0/2790935359 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:09.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319957854 0 0) 0x7feca802afa0 con 0x7feca802b4c0 2024-03-29T11:21:09.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.178 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.201+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec98003520 con 0x7feca802b4c0 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301485719 0 0) 0x7feca802c830 con 0x7feca8006c70 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:21:09.179 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.202+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca802afa0 con 0x7feca8006c70 2024-03-29T11:21:09.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398617695 0 0) 0x7feca802aba0 con 0x7feca8006890 2024-03-29T11:21:09.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.181 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feca802c830 con 0x7feca8006890 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295649763 0 0) 0x7feca802afa0 con 0x7feca8006c70 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 >> v1:172.21.15.3:6790/0 conn(0x7feca8006890 legacy=0x7feca80c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 >> v1:172.21.15.32:6789/0 conn(0x7feca802b4c0 legacy=0x7feca80cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca802ad30 con 0x7feca8006c70 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:09.182 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feca40014c0 con 0x7feca8006c70 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.204+0000 7fecadffb640 10 monclient: dump: 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.183 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feca4001620 con 0x7feca8006c70 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 1 -- 172.21.15.3:0/2790935359 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7feca4001a30 con 0x7feca8006c70 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecadffb640 10 monclient: dump: 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:09.184 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:09.185 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:09.185 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.185 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecaffff640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecaffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.205+0000 7fecaffff640 4 set_mon_vals no callback set 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.206+0000 7fecaffff640 10 monclient: shutdown 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.206+0000 7fecaffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.206+0000 7fecaffff640 1 -- 172.21.15.3:0/2790935359 >> v1:172.21.15.3:6789/0 conn(0x7feca8006c70 legacy=0x7feca80c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.206+0000 7fecaffff640 1 -- 172.21.15.3:0/2790935359 shutdown_connections 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.206+0000 7fecaffff640 1 -- 172.21.15.3:0/2790935359 >> 172.21.15.3:0/2790935359 conn(0x7feca80058f0 msgr2=0x7feca8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.207+0000 7fecaffff640 1 -- 172.21.15.3:0/2790935359 shutdown_connections 2024-03-29T11:21:09.186 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.207+0000 7fecaffff640 1 -- 172.21.15.3:0/2790935359 wait complete. 2024-03-29T11:21:09.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.208+0000 7fecaffff640 10 monclient: build_initial_monmap 2024-03-29T11:21:09.190 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.208+0000 7fecaffff640 10 monclient: monmap: 2024-03-29T11:21:09.190 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:09.190 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.209+0000 7fecaffff640 1 librados: starting msgr at 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.209+0000 7fecaffff640 1 librados: starting objecter 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.209+0000 7fecaffff640 1 Processor -- start 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 -- start start 2024-03-29T11:21:09.191 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 librados: setting wanted keys 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 librados: calling monclient init 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient: init 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-a con 0x7feca8006890 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-c con 0x7feca802b4c0 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient(hunting): picked mon.noname-b con 0x7feca80c43c0 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca802e1a0 con 0x7feca8006890 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca802cd90 con 0x7feca80c43c0 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca80c3a90 con 0x7feca802b4c0 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.210+0000 7fecaffff640 10 monclient(hunting): authenticate will time out at 1196.464987s 2024-03-29T11:21:09.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.212+0000 7fecaf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7feca80c43c0 0x7feca816bd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:50100/0 (socket says 172.21.15.3:50100) 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.212+0000 7fecaf7fe640 1 -- 172.21.15.3:0/1399075053 learned_addr learned my addr 172.21.15.3:0/1399075053 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.212+0000 7fec977fe640 10 client.?.objecter ms_handle_connect 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.212+0000 7fec977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251176580 0 0) 0x7feca802cd90 con 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c004eb0 con 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903402424 0 0) 0x7fec9c004eb0 con 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feca802cd90 con 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fecb004fdd0 con 0x7feca80c43c0 2024-03-29T11:21:09.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): dump: 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.194 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.213+0000 7fec977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:09.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.214+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1592206085 0 0) 0x7feca802cd90 con 0x7feca80c43c0 2024-03-29T11:21:09.195 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.214+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 >> v1:172.21.15.3:6790/0 conn(0x7feca802b4c0 legacy=0x7feca8165160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 >> v1:172.21.15.3:6789/0 conn(0x7feca8006890 legacy=0x7feca8164a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 1 monclient: found mon.b 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca80c3c20 con 0x7feca80c43c0 2024-03-29T11:21:09.197 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 client.?.objecter ms_handle_connect 0x7feca8006890 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fec977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 10 monclient: _renew_subs 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca80c15c0 con 0x7feca80c43c0 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 10 monclient: _renew_subs 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feca816d620 con 0x7feca80c43c0 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.215+0000 7fecaffff640 1 librados: init done 2024-03-29T11:21:09.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.217+0000 7fecaffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.217+0000 7fecaffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.217+0000 7fecaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.217+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca80c7b40 con 0x7feca80c43c0 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecb004f430 con 0x7feca80c43c0 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fecb004eee0 con 0x7feca80c43c0 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:09.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 10 monclient: dump: 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:09.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.218+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fecb0067be0 con 0x7feca80c43c0 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fecad7fa640 4 set_mon_vals no callback set 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fecb00ff650 con 0x7feca80c43c0 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fec977fe640 10 client.4474.objecter ms_dispatch 0x7fecb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fec977fe640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fec977fe640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.219+0000 7fec977fe640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.220+0000 7fecaeffd640 10 monclient: get_auth_request con 0x7fec9c06d9d0 auth_method 0 2024-03-29T11:21:09.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.220+0000 7fec977fe640 10 client.4474.objecter ms_handle_connect 0x7fec9c06d9d0 2024-03-29T11:21:09.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.226+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecb00cb060 con 0x7feca80c43c0 2024-03-29T11:21:09.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.226+0000 7fec977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:09.202 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.226+0000 7fec977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:09.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.552+0000 7fecaffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.552+0000 7fecaffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.552+0000 7fecaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:09.528 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.552+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feca802a520 con 0x7feca80c43c0 2024-03-29T11:21:09.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.553+0000 7fec977fe640 1 -- 172.21.15.3:0/1399075053 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fecb004f040 con 0x7feca80c43c0 2024-03-29T11:21:09.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.553+0000 7fec977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:09.530 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.553+0000 7fec977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:09.530 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:09.530 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 10 librados: watch_flush enter 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 10 librados: watch_flush exit 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 10 client.4474.objecter successfully canceled tick 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 >> v1:172.21.15.3:6816/844689376 conn(0x7fec9c06d9d0 legacy=0x7fec9c06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 10 monclient: shutdown 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.556+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 >> v1:172.21.15.32:6789/0 conn(0x7feca80c43c0 legacy=0x7feca816bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.557+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 shutdown_connections 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.557+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 >> 172.21.15.3:0/1399075053 conn(0x7feca80058f0 msgr2=0x7feca8038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.557+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 shutdown_connections 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.557+0000 7fecaffff640 1 -- 172.21.15.3:0/1399075053 wait complete. 2024-03-29T11:21:09.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:09.558+0000 7fecaffff640 1 librados: shutdown 2024-03-29T11:21:09.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:16.357 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.536+0000 7f0f515b6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.536+0000 7f0f515b6640 10 monclient: build_initial_monmap 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.536+0000 7f0f515b6640 10 monclient: monmap: 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:16.536931+0000 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:16.536931+0000 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:16.513 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.537+0000 7f0f515b6640 1 Processor -- start 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.537+0000 7f0f515b6640 1 -- start start 2024-03-29T11:21:16.514 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.537+0000 7f0f515b6640 10 monclient: init 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.537+0000 7f0f515b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.537+0000 7f0f515b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c0b4f20 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c0a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0aabd0 con 0x7f0f4c0b4f20 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0aaeb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.515 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c0ab190 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f515b6640 10 monclient(hunting): authenticate will time out at 1203.792082s 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f4a7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f0f4c0b4f20 0x7f0f4c1455f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:50566/0 (socket says 172.21.15.3:50566) 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f4a7fc640 1 -- 172.21.15.3:0/2042823979 learned_addr learned my addr 172.21.15.3:0/2042823979 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636575143 0 0) 0x7f0f4c0ab190 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f38003520 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195058670 0 0) 0x7f0f4c0aaeb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.516 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.538+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c0ab190 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541456986 0 0) 0x7f0f4c0aabd0 con 0x7f0f4c0b4f20 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:21:16.517 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f38003a20 con 0x7f0f4c0b4f20 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4177167100 0 0) 0x7f0f38003520 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 >> v1:172.21.15.32:6789/0 conn(0x7f0f4c0b4b40 legacy=0x7f0f4c13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 >> v1:172.21.15.3:6789/0 conn(0x7f0f4c0b4f20 legacy=0x7f0f4c1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c0b45a0 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:16.519 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f515b6640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f515b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f44001a10 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: dump: 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:16.520 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f515b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f44001bf0 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f515b6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.539+0000 7f0f515b6640 4 set_mon_vals no callback set 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f49ffb640 1 -- 172.21.15.3:0/2042823979 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f44001a10 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f49ffb640 10 monclient: dump: 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.521 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 10 monclient: shutdown 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 1 -- 172.21.15.3:0/2042823979 >> v1:172.21.15.3:6790/0 conn(0x7f0f4c0a9a90 legacy=0x7f0f4c145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 1 -- 172.21.15.3:0/2042823979 shutdown_connections 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 1 -- 172.21.15.3:0/2042823979 >> 172.21.15.3:0/2042823979 conn(0x7f0f4c066a00 msgr2=0x7f0f4c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.540+0000 7f0f515b6640 1 -- 172.21.15.3:0/2042823979 shutdown_connections 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.541+0000 7f0f515b6640 1 -- 172.21.15.3:0/2042823979 wait complete. 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.541+0000 7f0f515b6640 10 monclient: build_initial_monmap 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.541+0000 7f0f515b6640 10 monclient: monmap: 2024-03-29T11:21:16.522 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 librados: starting msgr at 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 librados: starting objecter 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 Processor -- start 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 -- start start 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 librados: setting wanted keys 2024-03-29T11:21:16.523 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 1 librados: calling monclient init 2024-03-29T11:21:16.524 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.542+0000 7f0f515b6640 10 monclient: init 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f4c0a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f4c0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f4c13c520 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c141d40 con 0x7f0f4c13c520 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.543+0000 7f0f515b6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c141fb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f515b6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4c137fd0 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f515b6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f515b6640 10 monclient(hunting): authenticate will time out at 1203.798011s 2024-03-29T11:21:16.531 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f4a7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f0f4c0b4b40 0x7f0f4c138a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:48820/0 (socket says 172.21.15.3:48820) 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f4a7fc640 1 -- 172.21.15.3:0/3282163012 learned_addr learned my addr 172.21.15.3:0/3282163012 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 client.?.objecter ms_handle_connect 0x7f0f4c13c520 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 client.?.objecter ms_handle_connect 0x7f0f4c0b4b40 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859251685 0 0) 0x7f0f4c141d40 con 0x7f0f4c13c520 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f2c004eb0 con 0x7f0f4c13c520 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481775530 0 0) 0x7f0f4c141fb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.544+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c141d40 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.545+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705396214 0 0) 0x7f0f2c004eb0 con 0x7f0f4c13c520 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.545+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f4c141fb0 con 0x7f0f4c13c520 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 client.?.objecter ms_handle_connect 0x7f0f4c0a9a90 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f44001f00 con 0x7f0f4c13c520 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): dump: 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.533 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1594125367 0 0) 0x7f0f4c141d40 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f2c004eb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f34001880 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): dump: 2024-03-29T11:21:16.534 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835912807 0 0) 0x7f0f4c137fd0 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:21:16.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c141d40 con 0x7f0f4c0a9a90 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 419069223 0 0) 0x7f0f2c004eb0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.546+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 >> v1:172.21.15.3:6790/0 conn(0x7f0f4c0a9a90 legacy=0x7f0f4c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 >> v1:172.21.15.3:6789/0 conn(0x7f0f4c13c520 legacy=0x7f0f4c1dda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 1 monclient: found mon.b 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c141100 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 10 monclient: _renew_subs 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4c142dc0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 10 monclient: _renew_subs 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4c139360 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f515b6640 1 librados: init done 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f340019e0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0f34004820 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f497fa640 4 set_mon_vals no callback set 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.547+0000 7f0f437fe640 10 monclient: dump: 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:16.537 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.549+0000 7f0f515b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.549+0000 7f0f515b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.549+0000 7f0f515b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.549+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30005080 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.551+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f0f3401a7f0 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.551+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f0f34084840 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f4affd640 10 monclient: get_auth_request con 0x7f0f2c06dfd0 auth_method 0 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f437fe640 10 client.4480.objecter ms_dispatch 0x7f0f4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f437fe640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f437fe640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f437fe640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.552+0000 7f0f437fe640 10 client.4480.objecter ms_handle_connect 0x7f0f2c06dfd0 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.559+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f3404ff50 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.559+0000 7f0f437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:16.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.559+0000 7f0f437fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:16.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.956+0000 7f0f515b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.956+0000 7f0f515b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.956+0000 7f0f515b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:16.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.956+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f30005330 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.957+0000 7f0f437fe640 1 -- 172.21.15.3:0/3282163012 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f0f34052050 con 0x7f0f4c0b4b40 2024-03-29T11:21:16.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.958+0000 7f0f437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:16.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.958+0000 7f0f437fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:16.934 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:16.935 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:16.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 10 librados: watch_flush enter 2024-03-29T11:21:16.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 10 librados: watch_flush exit 2024-03-29T11:21:16.937 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-29T11:21:16.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 10 client.4480.objecter successfully canceled tick 2024-03-29T11:21:16.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 >> v1:172.21.15.3:6816/844689376 conn(0x7f0f2c06dfd0 legacy=0x7f0f2c070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 10 monclient: shutdown 2024-03-29T11:21:16.938 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 >> v1:172.21.15.32:6789/0 conn(0x7f0f4c0b4b40 legacy=0x7f0f4c138a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.961+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 shutdown_connections 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.962+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 >> 172.21.15.3:0/3282163012 conn(0x7f0f4c066a00 msgr2=0x7f0f4c0b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.962+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 shutdown_connections 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.962+0000 7f0f515b6640 1 -- 172.21.15.3:0/3282163012 wait complete. 2024-03-29T11:21:16.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:16.962+0000 7f0f515b6640 1 librados: shutdown 2024-03-29T11:21:16.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:23.760 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:23.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.919+0000 7fdb2c255640 10 monclient: get_monmap_and_config 2024-03-29T11:21:23.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.919+0000 7fdb2c255640 10 monclient: build_initial_monmap 2024-03-29T11:21:23.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.919+0000 7fdb2c255640 10 monclient: monmap: 2024-03-29T11:21:23.897 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:23.897 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:23.897 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:23.920542+0000 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:23.920542+0000 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:23.898 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.920+0000 7fdb2c255640 1 Processor -- start 2024-03-29T11:21:23.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.920+0000 7fdb2c255640 1 -- start start 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.920+0000 7fdb2c255640 10 monclient: init 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.921+0000 7fdb2c255640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.921+0000 7fdb2c255640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.921+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240b4b40 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.921+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-a con 0x7fdb240b4f20 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.921+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240a9a90 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb2c255640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240aabd0 con 0x7fdb240b4f20 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb2c255640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240aaeb0 con 0x7fdb240a9a90 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb2c255640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb240ab190 con 0x7fdb240b4b40 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb2c255640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb2c255640 10 monclient(hunting): authenticate will time out at 1211.176066s 2024-03-29T11:21:23.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb29ff1640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdb240b4b40 0x7fdb2413be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:34832/0 (socket says 172.21.15.3:34832) 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb29ff1640 1 -- 172.21.15.3:0/4208095982 learned_addr learned my addr 172.21.15.3:0/4208095982 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478441156 0 0) 0x7fdb240ab190 con 0x7fdb240b4b40 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 10 monclient(hunting): _negotiate 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb10003520 con 0x7fdb240b4b40 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.922+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1869598275 0 0) 0x7fdb10003520 con 0x7fdb240b4b40 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.923+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 >> v1:172.21.15.32:6789/0 conn(0x7fdb240a9a90 legacy=0x7fdb24145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.923+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 >> v1:172.21.15.3:6789/0 conn(0x7fdb240b4f20 legacy=0x7fdb241455f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: _finish_hunting 0 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 1 monclient: found mon.noname-b 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:23.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb240b45a0 con 0x7fdb240b4b40 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: _finish_auth 0 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: _check_auth_tickets 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb20001690 con 0x7fdb240b4b40 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: dump: 2024-03-29T11:21:23.905 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb2c255640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb2c255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb20001860 con 0x7fdb240b4b40 2024-03-29T11:21:23.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 1 -- 172.21.15.3:0/4208095982 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb20001cb0 con 0x7fdb240b4b40 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb28fef640 10 monclient: dump: 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb2c255640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.924+0000 7fdb2c255640 4 set_mon_vals no callback set 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.925+0000 7fdb2c255640 10 monclient: shutdown 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.925+0000 7fdb2c255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.925+0000 7fdb2c255640 1 -- 172.21.15.3:0/4208095982 >> v1:172.21.15.3:6790/0 conn(0x7fdb240b4b40 legacy=0x7fdb2413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.926+0000 7fdb2c255640 1 -- 172.21.15.3:0/4208095982 shutdown_connections 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.926+0000 7fdb2c255640 1 -- 172.21.15.3:0/4208095982 >> 172.21.15.3:0/4208095982 conn(0x7fdb24066a00 msgr2=0x7fdb24066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.926+0000 7fdb2c255640 1 -- 172.21.15.3:0/4208095982 shutdown_connections 2024-03-29T11:21:23.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.926+0000 7fdb2c255640 1 -- 172.21.15.3:0/4208095982 wait complete. 2024-03-29T11:21:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.927+0000 7fdb2c255640 10 monclient: build_initial_monmap 2024-03-29T11:21:23.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.927+0000 7fdb2c255640 10 monclient: monmap: 2024-03-29T11:21:23.913 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 librados: starting msgr at 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 librados: starting objecter 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 Processor -- start 2024-03-29T11:21:23.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 -- start start 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 librados: setting wanted keys 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 1 librados: calling monclient init 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.928+0000 7fdb2c255640 10 monclient: init 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-b con 0x7fdb240a9a90 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-c con 0x7fdb240b4b40 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient(hunting): picked mon.noname-a con 0x7fdb2413f370 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb24131fe0 con 0x7fdb2413f370 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb241dbc60 con 0x7fdb240a9a90 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb241307e0 con 0x7fdb240b4b40 2024-03-29T11:21:23.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb2c255640 10 monclient(hunting): authenticate will time out at 1211.183672s 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb297f0640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fdb240b4b40 0x7fdb241dae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:35130/0 (socket says 172.21.15.3:35130) 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.929+0000 7fdb297f0640 1 -- 172.21.15.3:0/2185059219 learned_addr learned my addr 172.21.15.3:0/2185059219 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdb240b4b40 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432571068 0 0) 0x7fdb241307e0 con 0x7fdb240b4b40 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb08004eb0 con 0x7fdb240b4b40 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4010635178 0 0) 0x7fdb08004eb0 con 0x7fdb240b4b40 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.930+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb241307e0 con 0x7fdb240b4b40 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdb240a9a90 2024-03-29T11:21:23.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb14001960 con 0x7fdb240b4b40 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 241623212 0 0) 0x7fdb241307e0 con 0x7fdb240b4b40 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 >> v1:172.21.15.32:6789/0 conn(0x7fdb240a9a90 legacy=0x7fdb24130ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.931+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 >> v1:172.21.15.3:6789/0 conn(0x7fdb2413f370 legacy=0x7fdb241db550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 1 monclient: found mon.c 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb241313f0 con 0x7fdb240b4b40 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 10 monclient: _renew_subs 2024-03-29T11:21:23.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb241322c0 con 0x7fdb240b4b40 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 10 monclient: _renew_subs 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.932+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2412ff40 con 0x7fdb240b4b40 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb2c255640 1 librados: init done 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb14001c80 con 0x7fdb240b4b40 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fdb14003a40 con 0x7fdb240b4b40 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1a7fc640 10 monclient: dump: 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:23.919 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.933+0000 7fdb1bfff640 4 set_mon_vals no callback set 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.934+0000 7fdb2c255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.934+0000 7fdb2c255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.934+0000 7fdb2c255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.934+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c005080 con 0x7fdb240b4b40 2024-03-29T11:21:23.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.936+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fdb14003ba0 con 0x7fdb240b4b40 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.937+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fdb14084a90 con 0x7fdb240b4b40 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.937+0000 7fdb1a7fc640 10 client.4493.objecter ms_dispatch 0x7fdb24000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.937+0000 7fdb1a7fc640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.937+0000 7fdb1a7fc640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.937+0000 7fdb1a7fc640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.938+0000 7fdb29ff1640 10 monclient: get_auth_request con 0x7fdb0806dc60 auth_method 0 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.939+0000 7fdb1a7fc640 10 client.4493.objecter ms_handle_connect 0x7fdb0806dc60 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.943+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb14050370 con 0x7fdb240b4b40 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.943+0000 7fdb1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:23.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:23.943+0000 7fdb1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:24.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.261+0000 7fdb2c255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.261+0000 7fdb2c255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.261+0000 7fdb2c255640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:24.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.261+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb0c005330 con 0x7fdb240b4b40 2024-03-29T11:21:24.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.262+0000 7fdb1a7fc640 1 -- 172.21.15.3:0/2185059219 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fdb140522a0 con 0x7fdb240b4b40 2024-03-29T11:21:24.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.262+0000 7fdb1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:24.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.262+0000 7fdb1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:24.239 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:24.239 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 10 librados: watch_flush enter 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 10 librados: watch_flush exit 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 >> v1:172.21.15.3:6816/844689376 conn(0x7fdb0806dc60 legacy=0x7fdb08070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 10 monclient: shutdown 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 >> v1:172.21.15.3:6790/0 conn(0x7fdb240b4b40 legacy=0x7fdb241dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:24.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 shutdown_connections 2024-03-29T11:21:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.265+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 >> 172.21.15.3:0/2185059219 conn(0x7fdb24066a00 msgr2=0x7fdb24066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.266+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 shutdown_connections 2024-03-29T11:21:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.266+0000 7fdb2c255640 1 -- 172.21.15.3:0/2185059219 wait complete. 2024-03-29T11:21:24.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:24.266+0000 7fdb2c255640 1 librados: shutdown 2024-03-29T11:21:24.257 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:31.063 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:31.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.233+0000 7f6553fff640 10 monclient: get_monmap_and_config 2024-03-29T11:21:31.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.233+0000 7f6553fff640 10 monclient: build_initial_monmap 2024-03-29T11:21:31.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.233+0000 7f6553fff640 10 monclient: monmap: 2024-03-29T11:21:31.210 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:31.210 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:31.235044+0000 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:31.235044+0000 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:31.211 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.235+0000 7f6553fff640 1 Processor -- start 2024-03-29T11:21:31.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.235+0000 7f6553fff640 1 -- start start 2024-03-29T11:21:31.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.235+0000 7f6553fff640 10 monclient: init 2024-03-29T11:21:31.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.236+0000 7f6553fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:31.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.236+0000 7f6553fff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:31.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-c con 0x7f65540743e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-a con 0x7f65540747c0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-b con 0x7f6554146e60 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554143e70 con 0x7f65540747c0 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554146940 con 0x7f65540743e0 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540af6b0 con 0x7f6554146e60 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:31.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.237+0000 7f6553fff640 10 monclient(hunting): authenticate will time out at 1218.492766s 2024-03-29T11:21:31.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.240+0000 7f65527fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f65540747c0 0x7f65540aefa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34453/0 (socket says 172.21.15.3:34453) 2024-03-29T11:21:31.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f65527fc640 1 -- 172.21.15.3:0/3414615423 learned_addr learned my addr 172.21.15.3:0/3414615423 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:31.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424936334 0 0) 0x7f6554143e70 con 0x7f65540747c0 2024-03-29T11:21:31.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.241+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538003520 con 0x7f65540747c0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935593197 0 0) 0x7f6538003520 con 0x7f65540747c0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 >> v1:172.21.15.3:6790/0 conn(0x7f6554146e60 legacy=0x7f6554151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 >> v1:172.21.15.32:6789/0 conn(0x7f65540743e0 legacy=0x7f6554143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 monclient: found mon.noname-a 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554072fd0 con 0x7f65540747c0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:31.220 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6540001690 con 0x7f65540747c0 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6553fff640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6553fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: dump: 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.221 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6553fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6540001860 con 0x7f65540747c0 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 1 -- 172.21.15.3:0/3414615423 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6540001cb0 con 0x7f65540747c0 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6551ffb640 10 monclient: dump: 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.222 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6553fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:31.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.242+0000 7f6553fff640 4 set_mon_vals no callback set 2024-03-29T11:21:31.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 10 monclient: shutdown 2024-03-29T11:21:31.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 1 -- 172.21.15.3:0/3414615423 >> v1:172.21.15.3:6789/0 conn(0x7f65540747c0 legacy=0x7f65540aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 1 -- 172.21.15.3:0/3414615423 shutdown_connections 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 1 -- 172.21.15.3:0/3414615423 >> 172.21.15.3:0/3414615423 conn(0x7f65540b45a0 msgr2=0x7f65540b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 1 -- 172.21.15.3:0/3414615423 shutdown_connections 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.243+0000 7f6553fff640 1 -- 172.21.15.3:0/3414615423 wait complete. 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.244+0000 7f6553fff640 10 monclient: build_initial_monmap 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.244+0000 7f6553fff640 10 monclient: monmap: 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.225 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.246+0000 7f6553fff640 1 librados: starting msgr at 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.246+0000 7f6553fff640 1 librados: starting objecter 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.246+0000 7f6553fff640 1 Processor -- start 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.247+0000 7f6553fff640 1 -- start start 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.247+0000 7f6553fff640 1 librados: setting wanted keys 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.247+0000 7f6553fff640 1 librados: calling monclient init 2024-03-29T11:21:31.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.247+0000 7f6553fff640 10 monclient: init 2024-03-29T11:21:31.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.248+0000 7f6553fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:31.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.248+0000 7f6553fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:31.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.248+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-a con 0x7f65540743e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:31.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.248+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-c con 0x7f6554146e60 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f65527fc640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f6554146e60 0x7f655412ca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52700/0 (socket says 172.21.15.3:52700) 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f65537fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f655412d700 0x7f65541eef60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:33797/0 (socket says 172.21.15.3:33797) 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f65537fe640 1 -- 172.21.15.3:0/3747760694 learned_addr learned my addr 172.21.15.3:0/3747760694 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f6553fff640 10 monclient(hunting): picked mon.noname-b con 0x7f655412d700 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f6553fff640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554147fd0 con 0x7f65540743e0 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f6553fff640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540ad320 con 0x7f655412d700 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f6553fff640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540ad600 con 0x7f6554146e60 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f654b7fe640 10 client.?.objecter ms_handle_connect 0x7f655412d700 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f654b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f654b7fe640 10 client.?.objecter ms_handle_connect 0x7f6554146e60 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.252+0000 7f654b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.231 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f6553fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f6553fff640 10 monclient(hunting): authenticate will time out at 1218.508141s 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174723693 0 0) 0x7f65540ad600 con 0x7f6554146e60 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.253+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f6554146e60 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.254+0000 7f654b7fe640 10 client.?.objecter ms_handle_connect 0x7f65540743e0 2024-03-29T11:21:31.232 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.254+0000 7f654b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:31.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 387987646 0 0) 0x7f6554147fd0 con 0x7f65540743e0 2024-03-29T11:21:31.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65540ad600 con 0x7f65540743e0 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598604433 0 0) 0x7f653c004eb0 con 0x7f6554146e60 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6554147fd0 con 0x7f6554146e60 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594357334 0 0) 0x7f65540ad320 con 0x7f655412d700 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:21:31.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f653c004eb0 con 0x7f655412d700 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6540001b70 con 0x7f6554146e60 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:31.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 639325634 0 0) 0x7f653c004eb0 con 0x7f655412d700 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65540ad320 con 0x7f655412d700 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f654c001990 con 0x7f655412d700 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.238 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2315769411 0 0) 0x7f6554147fd0 con 0x7f6554146e60 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 >> v1:172.21.15.32:6789/0 conn(0x7f655412d700 legacy=0x7f65541eef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 >> v1:172.21.15.3:6789/0 conn(0x7f65540743e0 legacy=0x7f65540acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 monclient: found mon.c 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:31.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540adb20 con 0x7f6554146e60 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.257+0000 7f654b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 10 monclient: _renew_subs 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65540add70 con 0x7f6554146e60 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 10 monclient: _renew_subs 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65541319d0 con 0x7f6554146e60 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.258+0000 7f6553fff640 1 librados: init done 2024-03-29T11:21:31.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.260+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6540001010 con 0x7f6554146e60 2024-03-29T11:21:31.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.260+0000 7f654b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:31.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.260+0000 7f6550ff9640 4 set_mon_vals no callback set 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.261+0000 7f65497fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6540004d90 con 0x7f6554146e60 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f65497fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f65497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6530005080 con 0x7f6554146e60 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f654b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f654b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f654b7fe640 10 monclient: dump: 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:31.243 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.262+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f654001ad60 con 0x7f6554146e60 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f6540084bc0 con 0x7f6554146e60 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f6552ffd640 10 monclient: get_auth_request con 0x7f653c06e110 auth_method 0 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f654b7fe640 10 client.4496.objecter ms_dispatch 0x7f6554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f654b7fe640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f654b7fe640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.263+0000 7f654b7fe640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T11:21:31.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.264+0000 7f654b7fe640 10 client.4496.objecter ms_handle_connect 0x7f653c06e110 2024-03-29T11:21:31.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.270+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6540050520 con 0x7f6554146e60 2024-03-29T11:21:31.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.270+0000 7f654b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:31.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.270+0000 7f654b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:31.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.602+0000 7f65497fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.602+0000 7f65497fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.602+0000 7f65497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.602+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6530005330 con 0x7f6554146e60 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.603+0000 7f654b7fe640 1 -- 172.21.15.3:0/3747760694 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f6540052450 con 0x7f6554146e60 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.603+0000 7f654b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.603+0000 7f654b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:31.581 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:31.582 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:31.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.607+0000 7f65497fa640 10 librados: watch_flush enter 2024-03-29T11:21:31.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.607+0000 7f65497fa640 10 librados: watch_flush exit 2024-03-29T11:21:31.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.608+0000 7f65497fa640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T11:21:31.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.608+0000 7f65497fa640 10 client.4496.objecter successfully canceled tick 2024-03-29T11:21:31.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 >> v1:172.21.15.3:6816/844689376 conn(0x7f653c06e110 legacy=0x7f653c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 10 monclient: shutdown 2024-03-29T11:21:31.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:31.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 >> v1:172.21.15.3:6790/0 conn(0x7f6554146e60 legacy=0x7f655412ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:31.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 shutdown_connections 2024-03-29T11:21:31.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 >> 172.21.15.3:0/3747760694 conn(0x7f65540b45a0 msgr2=0x7f65540aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:31.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 shutdown_connections 2024-03-29T11:21:31.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 -- 172.21.15.3:0/3747760694 wait complete. 2024-03-29T11:21:31.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:31.609+0000 7f65497fa640 1 librados: shutdown 2024-03-29T11:21:31.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:38.410 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.585+0000 7f9f8ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:21:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.585+0000 7f9f8ffff640 10 monclient: build_initial_monmap 2024-03-29T11:21:38.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.585+0000 7f9f8ffff640 10 monclient: monmap: 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:38.586920+0000 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:38.586920+0000 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:38.564 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.587+0000 7f9f8ffff640 1 Processor -- start 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.587+0000 7f9f8ffff640 1 -- start start 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.587+0000 7f9f8ffff640 10 monclient: init 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.587+0000 7f9f8ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f9f88034850 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:38.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9f8802a640 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:38.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f9f880327d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8802c820 con 0x7f9f8802a640 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8802a120 con 0x7f9f88034850 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f880c75b0 con 0x7f9f880327d0 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.588+0000 7f9f8ffff640 10 monclient(hunting): authenticate will time out at 1225.843098s 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8f7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f9f880327d0 0x7f9f880cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53785/0 (socket says 172.21.15.3:53785) 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8f7fe640 1 -- 172.21.15.3:0/720998240 learned_addr learned my addr 172.21.15.3:0/720998240 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918163177 0 0) 0x7f9f880c75b0 con 0x7f9f880327d0 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.594+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f78003520 con 0x7f9f880327d0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179826923 0 0) 0x7f9f78003520 con 0x7f9f880327d0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 >> v1:172.21.15.32:6789/0 conn(0x7f9f88034850 legacy=0x7f9f880c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 >> v1:172.21.15.3:6789/0 conn(0x7f9f8802a640 legacy=0x7f9f880320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88033110 con 0x7f9f880327d0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9f84001690 con 0x7f9f880327d0 2024-03-29T11:21:38.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: dump: 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8ffff640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:38.576 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f84001860 con 0x7f9f880327d0 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.595+0000 7f9f8ffff640 4 set_mon_vals no callback set 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8dffb640 1 -- 172.21.15.3:0/720998240 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9f84001cb0 con 0x7f9f880327d0 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8dffb640 10 monclient: dump: 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:38.577 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:38.578 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8ffff640 10 monclient: shutdown 2024-03-29T11:21:38.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:38.578 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.596+0000 7f9f8ffff640 1 -- 172.21.15.3:0/720998240 >> v1:172.21.15.3:6790/0 conn(0x7f9f880327d0 legacy=0x7f9f880cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.598+0000 7f9f8ffff640 1 -- 172.21.15.3:0/720998240 shutdown_connections 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.598+0000 7f9f8ffff640 1 -- 172.21.15.3:0/720998240 >> 172.21.15.3:0/720998240 conn(0x7f9f880058f0 msgr2=0x7f9f88005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.598+0000 7f9f8ffff640 1 -- 172.21.15.3:0/720998240 shutdown_connections 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.598+0000 7f9f8ffff640 1 -- 172.21.15.3:0/720998240 wait complete. 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.600+0000 7f9f8ffff640 10 monclient: build_initial_monmap 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.600+0000 7f9f8ffff640 10 monclient: monmap: 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.579 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.601+0000 7f9f8ffff640 1 librados: starting msgr at 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.601+0000 7f9f8ffff640 1 librados: starting objecter 2024-03-29T11:21:38.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.601+0000 7f9f8ffff640 1 Processor -- start 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.603+0000 7f9f8ffff640 1 -- start start 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.603+0000 7f9f8ffff640 1 librados: setting wanted keys 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.603+0000 7f9f8ffff640 1 librados: calling monclient init 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.603+0000 7f9f8ffff640 10 monclient: init 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f9f8802a640 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f9f880327d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f9f88034850 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:38.581 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8816cc60 con 0x7f9f880327d0 2024-03-29T11:21:38.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f88152b10 con 0x7f9f8802a640 2024-03-29T11:21:38.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f88151a20 con 0x7f9f88034850 2024-03-29T11:21:38.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:38.582 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.604+0000 7f9f8ffff640 10 monclient(hunting): authenticate will time out at 1225.859204s 2024-03-29T11:21:38.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.607+0000 7f9f8f7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f9f88034850 0x7f9f880c0220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:53855/0 (socket says 172.21.15.3:53855) 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.607+0000 7f9f8f7fe640 1 -- 172.21.15.3:0/1988658445 learned_addr learned my addr 172.21.15.3:0/1988658445 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.607+0000 7f9f77fff640 10 client.?.objecter ms_handle_connect 0x7f9f88034850 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.607+0000 7f9f77fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125106438 0 0) 0x7f9f88151a20 con 0x7f9f88034850 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:21:38.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.608+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f7c004eb0 con 0x7f9f88034850 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901500154 0 0) 0x7f9f7c004eb0 con 0x7f9f88034850 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f88151a20 con 0x7f9f88034850 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9f84001b70 con 0x7f9f88034850 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 10 monclient(hunting): dump: 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.587 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1723811479 0 0) 0x7f9f88151a20 con 0x7f9f88034850 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 >> v1:172.21.15.32:6789/0 conn(0x7f9f8802a640 legacy=0x7f9f88151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.588 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.610+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 >> v1:172.21.15.3:6789/0 conn(0x7f9f880327d0 legacy=0x7f9f8816d0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:21:38.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:38.592 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 1 monclient: found mon.c 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8816d800 con 0x7f9f88034850 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 10 monclient: _finish_auth 0 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f77fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 10 monclient: _renew_subs 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f88150720 con 0x7f9f88034850 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 10 monclient: _renew_subs 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.593 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f880c2a50 con 0x7f9f88034850 2024-03-29T11:21:38.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.613+0000 7f9f8ffff640 1 librados: init done 2024-03-29T11:21:38.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.615+0000 7f9f8ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.615+0000 7f9f8ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.615+0000 7f9f8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.594 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.615+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8816ca30 con 0x7f9f88034850 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f84004bc0 con 0x7f9f88034850 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9f84004d30 con 0x7f9f88034850 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.616+0000 7f9f77fff640 10 monclient: dump: 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:38.595 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:38.596 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.618+0000 7f9f8cff9640 4 set_mon_vals no callback set 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.620+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f9f8401ad00 con 0x7f9f88034850 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.621+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f9f84084a60 con 0x7f9f88034850 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.621+0000 7f9f77fff640 10 client.4502.objecter ms_dispatch 0x7f9f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.621+0000 7f9f77fff640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.621+0000 7f9f77fff640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:38.598 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.621+0000 7f9f77fff640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T11:21:38.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.623+0000 7f9f8effd640 10 monclient: get_auth_request con 0x7f9f7c06dc40 auth_method 0 2024-03-29T11:21:38.602 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.623+0000 7f9f77fff640 10 client.4502.objecter ms_handle_connect 0x7f9f7c06dc40 2024-03-29T11:21:38.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.626+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f84050340 con 0x7f9f88034850 2024-03-29T11:21:38.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.626+0000 7f9f77fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:38.604 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.626+0000 7f9f77fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:38.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f8ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:38.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f8ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:38.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f8ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:21:38.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f88004cd0 con 0x7f9f88034850 2024-03-29T11:21:38.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f77fff640 1 -- 172.21.15.3:0/1988658445 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f9f84052270 con 0x7f9f88034850 2024-03-29T11:21:38.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f77fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:38.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.956+0000 7f9f77fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:38.937 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:38.937 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.960+0000 7f9f8ffff640 10 librados: watch_flush enter 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.960+0000 7f9f8ffff640 10 librados: watch_flush exit 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.960+0000 7f9f8ffff640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.960+0000 7f9f8ffff640 10 client.4502.objecter successfully canceled tick 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 >> v1:172.21.15.3:6816/844689376 conn(0x7f9f7c06dc40 legacy=0x7f9f7c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 10 monclient: shutdown 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 >> v1:172.21.15.3:6790/0 conn(0x7f9f88034850 legacy=0x7f9f880c0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 shutdown_connections 2024-03-29T11:21:38.939 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.961+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 >> 172.21.15.3:0/1988658445 conn(0x7f9f880058f0 msgr2=0x7f9f88029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:38.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.963+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 shutdown_connections 2024-03-29T11:21:38.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.963+0000 7f9f8ffff640 1 -- 172.21.15.3:0/1988658445 wait complete. 2024-03-29T11:21:38.940 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:38.963+0000 7f9f8ffff640 1 librados: shutdown 2024-03-29T11:21:38.957 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:45.764 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:45.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.935+0000 7f3122c3b640 10 monclient: get_monmap_and_config 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.935+0000 7f3122c3b640 10 monclient: build_initial_monmap 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.935+0000 7f3122c3b640 10 monclient: monmap: 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:45.936207+0000 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:45.936207+0000 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:45.912 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:45.913 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:45.913 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.936+0000 7f3122c3b640 1 Processor -- start 2024-03-29T11:21:45.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.936+0000 7f3122c3b640 1 -- start start 2024-03-29T11:21:45.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.936+0000 7f3122c3b640 10 monclient: init 2024-03-29T11:21:45.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.937+0000 7f3122c3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:45.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.937+0000 7f3122c3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:45.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.937+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-a con 0x7f31140389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:45.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.937+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-c con 0x7f3114038dc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:45.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3114030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:45.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31140c54c0 con 0x7f31140389e0 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3114036430 con 0x7f3114038dc0 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3114039840 con 0x7f3114030660 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.940+0000 7f3122c3b640 10 monclient(hunting): authenticate will time out at 1233.194942s 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.941+0000 7f3121c39640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f31140389e0 0x7f3114033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60381/0 (socket says 172.21.15.3:60381) 2024-03-29T11:21:45.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.941+0000 7f3121c39640 1 -- 172.21.15.3:0/2679629489 learned_addr learned my addr 172.21.15.3:0/2679629489 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:45.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128529825 0 0) 0x7f3114036430 con 0x7f3114038dc0 2024-03-29T11:21:45.922 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:21:45.923 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3108003520 con 0x7f3114038dc0 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255306451 0 0) 0x7f31140c54c0 con 0x7f31140389e0 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3114036430 con 0x7f31140389e0 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302568555 0 0) 0x7f3114039840 con 0x7f3114030660 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31140c54c0 con 0x7f3114030660 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3517501903 0 0) 0x7f3114036430 con 0x7f31140389e0 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.946+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 >> v1:172.21.15.3:6790/0 conn(0x7f3114030660 legacy=0x7f31140d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 >> v1:172.21.15.32:6789/0 conn(0x7f3114038dc0 legacy=0x7f31140cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: _finish_hunting 0 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 1 monclient: found mon.noname-a 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3114035690 con 0x7f31140389e0 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: _finish_auth 0 2024-03-29T11:21:45.927 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: _check_auth_tickets 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3118001670 con 0x7f31140389e0 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.947+0000 7f3120c37640 10 monclient: dump: 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:45.928 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:45.929 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:45.929 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.931 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31180017d0 con 0x7f31140389e0 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 1 -- 172.21.15.3:0/2679629489 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3118001ba0 con 0x7f31140389e0 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3120c37640 10 monclient: dump: 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:45.932 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3122c3b640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3122c3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.950+0000 7f3122c3b640 4 set_mon_vals no callback set 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.951+0000 7f3122c3b640 10 monclient: shutdown 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.951+0000 7f3122c3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.951+0000 7f3122c3b640 1 -- 172.21.15.3:0/2679629489 >> v1:172.21.15.3:6789/0 conn(0x7f31140389e0 legacy=0x7f3114033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.951+0000 7f3122c3b640 1 -- 172.21.15.3:0/2679629489 shutdown_connections 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.951+0000 7f3122c3b640 1 -- 172.21.15.3:0/2679629489 >> 172.21.15.3:0/2679629489 conn(0x7f3114005c00 msgr2=0x7f3114006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.952+0000 7f3122c3b640 1 -- 172.21.15.3:0/2679629489 shutdown_connections 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.952+0000 7f3122c3b640 1 -- 172.21.15.3:0/2679629489 wait complete. 2024-03-29T11:21:45.933 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.953+0000 7f3122c3b640 10 monclient: build_initial_monmap 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.953+0000 7f3122c3b640 10 monclient: monmap: 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:45.934 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 librados: starting msgr at 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 librados: starting objecter 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 Processor -- start 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 -- start start 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 librados: setting wanted keys 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 1 librados: calling monclient init 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.954+0000 7f3122c3b640 10 monclient: init 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.955+0000 7f3122c3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:45.935 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.955+0000 7f3122c3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:45.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.955+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-b con 0x7f3114030660 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:45.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.955+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-c con 0x7f31140389e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:45.936 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.959+0000 7f3121c39640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3114030660 0x7f3114163f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:52623/0 (socket says 172.21.15.3:52623) 2024-03-29T11:21:45.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f3121c39640 1 -- 172.21.15.3:0/760884960 learned_addr learned my addr 172.21.15.3:0/760884960 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:45.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f3122c3b640 10 monclient(hunting): picked mon.noname-a con 0x7f311402fbc0 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:45.949 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f31127fc640 10 client.?.objecter ms_handle_connect 0x7f3114030660 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f31127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f31127fc640 10 client.?.objecter ms_handle_connect 0x7f31140389e0 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.962+0000 7f31127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31140c3410 con 0x7f311402fbc0 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311402f6a0 con 0x7f3114030660 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f311402f2e0 con 0x7f31140389e0 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f3122c3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f3122c3b640 10 monclient(hunting): authenticate will time out at 1233.217539s 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730865301 0 0) 0x7f311402f2e0 con 0x7f31140389e0 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.950 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3100004eb0 con 0x7f31140389e0 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 client.?.objecter ms_handle_connect 0x7f311402fbc0 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.963+0000 7f31127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593467684 0 0) 0x7f31140c3410 con 0x7f311402fbc0 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:21:45.951 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f311402f2e0 con 0x7f311402fbc0 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209561729 0 0) 0x7f311402f6a0 con 0x7f3114030660 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31140c3410 con 0x7f3114030660 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273014381 0 0) 0x7f311402f2e0 con 0x7f311402fbc0 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f311402f6a0 con 0x7f311402fbc0 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f311c04fdd0 con 0x7f311402fbc0 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): dump: 2024-03-29T11:21:45.952 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2901010046 0 0) 0x7f311402f6a0 con 0x7f311402fbc0 2024-03-29T11:21:45.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 >> v1:172.21.15.3:6790/0 conn(0x7f31140389e0 legacy=0x7f31140c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 >> v1:172.21.15.32:6789/0 conn(0x7f3114030660 legacy=0x7f3114163f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 monclient: found mon.a 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f311402f470 con 0x7f311402fbc0 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.964+0000 7f31127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.965+0000 7f3122c3b640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:21:45.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.965+0000 7f3122c3b640 10 monclient: _renew_subs 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.965+0000 7f3122c3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.965+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31140300f0 con 0x7f311402fbc0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.966+0000 7f3122c3b640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.966+0000 7f3122c3b640 10 monclient: _renew_subs 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.966+0000 7f3122c3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.966+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f311402ed80 con 0x7f311402fbc0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.966+0000 7f3122c3b640 1 librados: init done 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f311c04f430 con 0x7f311402fbc0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f311c04eee0 con 0x7f311402fbc0 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:45.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31137fe640 4 set_mon_vals no callback set 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f31127fc640 10 monclient: dump: 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:45.956 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.967+0000 7f3122c3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.968+0000 7f3122c3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.968+0000 7f3122c3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.968+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3104005080 con 0x7f311402fbc0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.968+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f311c067be0 con 0x7f311402fbc0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.969+0000 7f3121c39640 10 monclient: get_auth_request con 0x7f310006dbe0 auth_method 0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 10 client.4476.objecter ms_handle_connect 0x7f310006dbe0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f311c0cd990 con 0x7f311402fbc0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 10 client.4476.objecter ms_dispatch 0x7f311c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:45.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.970+0000 7f31127fc640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-29T11:21:45.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.981+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f311c0ca330 con 0x7f311402fbc0 2024-03-29T11:21:45.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.981+0000 7f31127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:45.958 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:45.981+0000 7f31127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:46.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.317+0000 7f3122c3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.317+0000 7f3122c3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.317+0000 7f3122c3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:21:46.295 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.317+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3104005330 con 0x7f311402fbc0 2024-03-29T11:21:46.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.321+0000 7f31127fc640 1 -- 172.21.15.3:0/760884960 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f311c0ca4d0 con 0x7f311402fbc0 2024-03-29T11:21:46.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.321+0000 7f31127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:46.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.321+0000 7f31127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:46.299 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:46.299 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:46.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.325+0000 7f3122c3b640 10 librados: watch_flush enter 2024-03-29T11:21:46.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 10 librados: watch_flush exit 2024-03-29T11:21:46.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-29T11:21:46.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 10 client.4476.objecter successfully canceled tick 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 >> v1:172.21.15.3:6816/844689376 conn(0x7f310006dbe0 legacy=0x7f31000700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 10 monclient: shutdown 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 >> v1:172.21.15.3:6789/0 conn(0x7f311402fbc0 legacy=0x7f31140c2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 shutdown_connections 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 >> 172.21.15.3:0/760884960 conn(0x7f3114005c00 msgr2=0x7f3114034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 shutdown_connections 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 -- 172.21.15.3:0/760884960 wait complete. 2024-03-29T11:21:46.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:46.327+0000 7f3122c3b640 1 librados: shutdown 2024-03-29T11:21:46.320 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:53.127 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.301+0000 7f7cdfca8640 10 monclient: get_monmap_and_config 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.301+0000 7f7cdfca8640 10 monclient: build_initial_monmap 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.301+0000 7f7cdfca8640 10 monclient: monmap: 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:21:53.302090+0000 2024-03-29T11:21:53.278 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:21:53.302090+0000 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:21:53.279 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.302+0000 7f7cdfca8640 1 Processor -- start 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.302+0000 7f7cdfca8640 1 -- start start 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.302+0000 7f7cdfca8640 10 monclient: init 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:53.280 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd8149c10 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd80a9630 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd814a450 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80aad40 con 0x7f7cd814a450 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd814a8e0 con 0x7f7cd80a9630 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd80abb60 con 0x7f7cd8149c10 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:53.281 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.303+0000 7f7cdfca8640 10 monclient(hunting): authenticate will time out at 1240.557776s 2024-03-29T11:21:53.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.306+0000 7f7cde245640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7cd814a450 0x7f7cd8151d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:51724/0 (socket says 172.21.15.3:51724) 2024-03-29T11:21:53.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.306+0000 7f7cde245640 1 -- 172.21.15.3:0/2833794869 learned_addr learned my addr 172.21.15.3:0/2833794869 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:53.284 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711268444 0 0) 0x7f7cd80aad40 con 0x7f7cd814a450 2024-03-29T11:21:53.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:21:53.285 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc4003520 con 0x7f7cd814a450 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897019476 0 0) 0x7f7cd814a8e0 con 0x7f7cd80a9630 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd80aad40 con 0x7f7cd80a9630 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3028655901 0 0) 0x7f7cc4003520 con 0x7f7cd814a450 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 >> v1:172.21.15.3:6790/0 conn(0x7f7cd8149c10 legacy=0x7f7cd80ae7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 >> v1:172.21.15.32:6789/0 conn(0x7f7cd80a9630 legacy=0x7f7cd8143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient: _finish_hunting 0 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 monclient: found mon.noname-a 2024-03-29T11:21:53.286 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd80aa8d0 con 0x7f7cd814a450 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient: _finish_auth 0 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient: _check_auth_tickets 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdfca8640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdfca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7cd4001690 con 0x7f7cd814a450 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.308+0000 7f7cdca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: dump: 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:53.287 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdfca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd40017f0 con 0x7f7cd814a450 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdfca8640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdfca8640 4 set_mon_vals no callback set 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 1 -- 172.21.15.3:0/2833794869 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7cd4001c40 con 0x7f7cd814a450 2024-03-29T11:21:53.288 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.309+0000 7f7cdca42640 10 monclient: dump: 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:53.289 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.310+0000 7f7cdfca8640 10 monclient: shutdown 2024-03-29T11:21:53.290 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.310+0000 7f7cdfca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.310+0000 7f7cdfca8640 1 -- 172.21.15.3:0/2833794869 >> v1:172.21.15.3:6789/0 conn(0x7f7cd814a450 legacy=0x7f7cd8151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.310+0000 7f7cdfca8640 1 -- 172.21.15.3:0/2833794869 shutdown_connections 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.310+0000 7f7cdfca8640 1 -- 172.21.15.3:0/2833794869 >> 172.21.15.3:0/2833794869 conn(0x7f7cd80b3c10 msgr2=0x7f7cd80b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.311+0000 7f7cdfca8640 1 -- 172.21.15.3:0/2833794869 shutdown_connections 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.311+0000 7f7cdfca8640 1 -- 172.21.15.3:0/2833794869 wait complete. 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.312+0000 7f7cdfca8640 10 monclient: build_initial_monmap 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.312+0000 7f7cdfca8640 10 monclient: monmap: 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:21:53.291 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 librados: starting msgr at 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 librados: starting objecter 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 Processor -- start 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 -- start start 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 librados: setting wanted keys 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 1 librados: calling monclient init 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.313+0000 7f7cdfca8640 10 monclient: init 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-c con 0x7f7cd80a9630 addr v1:172.21.15.3:6790/0 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-b con 0x7f7cd8149c10 addr v1:172.21.15.32:6789/0 2024-03-29T11:21:53.292 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient(hunting): picked mon.noname-a con 0x7f7cd814a450 addr v1:172.21.15.3:6789/0 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd81db1f0 con 0x7f7cd814a450 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd81dbe20 con 0x7f7cd8149c10 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cd81dbfb0 con 0x7f7cd80a9630 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.314+0000 7f7cdfca8640 10 monclient(hunting): authenticate will time out at 1240.568192s 2024-03-29T11:21:53.293 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.316+0000 7f7cde245640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f7cd814a450 0x7f7cd81e5b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:55308/0 (socket says 172.21.15.3:55308) 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.316+0000 7f7cde245640 1 -- 172.21.15.3:0/1056014449 learned_addr learned my addr 172.21.15.3:0/1056014449 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd814a450 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877884770 0 0) 0x7f7cd81db1f0 con 0x7f7cd814a450 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:21:53.294 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cbc004eb0 con 0x7f7cd814a450 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 client.?.objecter ms_handle_connect 0x7f7cd8149c10 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.317+0000 7f7cce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049199248 0 0) 0x7f7cd81dbe20 con 0x7f7cd8149c10 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd81db1f0 con 0x7f7cd8149c10 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301483913 0 0) 0x7f7cd81db1f0 con 0x7f7cd8149c10 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cbc005490 con 0x7f7cd8149c10 2024-03-29T11:21:53.298 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7cd0001c70 con 0x7f7cd8149c10 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.299 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.318+0000 7f7cce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3911507580 0 0) 0x7f7cbc004eb0 con 0x7f7cd814a450 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cbc005b60 con 0x7f7cd814a450 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7cd4001b40 con 0x7f7cd814a450 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient(hunting): dump: 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.301 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754296084 0 0) 0x7f7cbc005490 con 0x7f7cd8149c10 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 >> v1:172.21.15.3:6790/0 conn(0x7f7cd80a9630 legacy=0x7f7cd81db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 >> v1:172.21.15.3:6789/0 conn(0x7f7cd814a450 legacy=0x7f7cd81e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 monclient: found mon.b 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd806f0c0 con 0x7f7cd8149c10 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:53.302 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 10 monclient: _renew_subs 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd8144c10 con 0x7f7cd8149c10 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 10 monclient: _renew_subs 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd806de80 con 0x7f7cd8149c10 2024-03-29T11:21:53.303 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.322+0000 7f7cdfca8640 1 librados: init done 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.324+0000 7f7cdfca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.325+0000 7f7cdfca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.325+0000 7f7cdfca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.325+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd81db380 con 0x7f7cd8149c10 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.326+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cd0003970 con 0x7f7cd8149c10 2024-03-29T11:21:53.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.326+0000 7f7cce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.326+0000 7f7ccffff640 4 set_mon_vals no callback set 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.327+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7cd0001c70 con 0x7f7cd8149c10 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.327+0000 7f7cce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.327+0000 7f7cce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.327+0000 7f7cce7fc640 10 monclient: dump: 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:21:53.305 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:21:53.306 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:21:53.306 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.328+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f7cd00199d0 con 0x7f7cd8149c10 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.329+0000 7f7cdda44640 10 monclient: get_auth_request con 0x7f7cbc06e390 auth_method 0 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.329+0000 7f7cce7fc640 10 client.4495.objecter ms_handle_connect 0x7f7cbc06e390 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.330+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f7cd0053690 con 0x7f7cd8149c10 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.330+0000 7f7cce7fc640 10 client.4495.objecter ms_dispatch 0x7f7cd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.330+0000 7f7cce7fc640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.330+0000 7f7cce7fc640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:53.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.330+0000 7f7cce7fc640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:21:53.310 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.334+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cd004f930 con 0x7f7cd8149c10 2024-03-29T11:21:53.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.334+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:53.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.334+0000 7f7cce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:53.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.672+0000 7f7cdfca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.672+0000 7f7cdfca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.672+0000 7f7cdfca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:21:53.649 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.672+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cd80aee80 con 0x7f7cd8149c10 2024-03-29T11:21:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.674+0000 7f7cce7fc640 1 -- 172.21.15.3:0/1056014449 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f7cd004f930 con 0x7f7cd8149c10 2024-03-29T11:21:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.674+0000 7f7cce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:53.650 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.674+0000 7f7cce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:53.651 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:21:53.652 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:53.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.678+0000 7f7cdfca8640 10 librados: watch_flush enter 2024-03-29T11:21:53.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.678+0000 7f7cdfca8640 10 librados: watch_flush exit 2024-03-29T11:21:53.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.678+0000 7f7cdfca8640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:21:53.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.678+0000 7f7cdfca8640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 >> v1:172.21.15.3:6816/844689376 conn(0x7f7cbc06e390 legacy=0x7f7cbc070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 10 monclient: shutdown 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 >> v1:172.21.15.32:6789/0 conn(0x7f7cd8149c10 legacy=0x7f7cd806e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 shutdown_connections 2024-03-29T11:21:53.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 >> 172.21.15.3:0/1056014449 conn(0x7f7cd80b3c10 msgr2=0x7f7cd813fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:53.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 shutdown_connections 2024-03-29T11:21:53.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 -- 172.21.15.3:0/1056014449 wait complete. 2024-03-29T11:21:53.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:21:53.682+0000 7f7cdfca8640 1 librados: shutdown 2024-03-29T11:21:53.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:00.485 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:00.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.657+0000 7f077e358640 10 monclient: get_monmap_and_config 2024-03-29T11:22:00.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.657+0000 7f077e358640 10 monclient: build_initial_monmap 2024-03-29T11:22:00.633 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.657+0000 7f077e358640 10 monclient: monmap: 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:00.658098+0000 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:00.658098+0000 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:00.634 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.658+0000 7f077e358640 1 Processor -- start 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.658+0000 7f077e358640 1 -- start start 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.658+0000 7f077e358640 10 monclient: init 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:00.638 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient(hunting): picked mon.noname-a con 0x7f07780b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient(hunting): picked mon.noname-c con 0x7f07780b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f07777fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f07780b2cb0 0x7f0778142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:35902/0 (socket says 172.21.15.3:35902) 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient(hunting): picked mon.noname-b con 0x7f07780b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780aab60 con 0x7f07780b28d0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780aae40 con 0x7f07780b2cb0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f07777fe640 1 -- 172.21.15.3:0/3258217703 learned_addr learned my addr 172.21.15.3:0/3258217703 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07780ab120 con 0x7f07780b3560 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.659+0000 7f077e358640 10 monclient(hunting): authenticate will time out at 1247.913624s 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113534046 0 0) 0x7f07780aae40 con 0x7f07780b2cb0 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.639 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0760003520 con 0x7f07780b2cb0 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434591957 0 0) 0x7f07780aab60 con 0x7f07780b28d0 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07780aae40 con 0x7f07780b28d0 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842641153 0 0) 0x7f0760003520 con 0x7f07780b2cb0 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.660+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 >> v1:172.21.15.3:6790/0 conn(0x7f07780b3560 legacy=0x7f07781428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 >> v1:172.21.15.3:6789/0 conn(0x7f07780b28d0 legacy=0x7f077813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 1 monclient: found mon.noname-c 2024-03-29T11:22:00.640 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077813c380 con 0x7f07780b2cb0 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f076c001690 con 0x7f07780b2cb0 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 10 monclient: dump: 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.641 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f077e358640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f077e358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.661+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f076c0017f0 con 0x7f07780b2cb0 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f0776ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f0776ffd640 1 -- 172.21.15.3:0/3258217703 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f076c001c00 con 0x7f07780b2cb0 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f0776ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f0776ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:00.642 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f0776ffd640 10 monclient: dump: 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f077e358640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:00.643 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f077e358640 4 set_mon_vals no callback set 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.662+0000 7f077e358640 10 monclient: shutdown 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 >> v1:172.21.15.32:6789/0 conn(0x7f07780b2cb0 legacy=0x7f0778142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 shutdown_connections 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 >> 172.21.15.3:0/3258217703 conn(0x7f0778066a00 msgr2=0x7f0778066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 shutdown_connections 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.663+0000 7f077e358640 1 -- 172.21.15.3:0/3258217703 wait complete. 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.664+0000 7f077e358640 10 monclient: build_initial_monmap 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.664+0000 7f077e358640 10 monclient: monmap: 2024-03-29T11:22:00.651 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 librados: starting msgr at 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 librados: starting objecter 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 Processor -- start 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 -- start start 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 librados: setting wanted keys 2024-03-29T11:22:00.652 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 1 librados: calling monclient init 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.665+0000 7f077e358640 10 monclient: init 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient(hunting): picked mon.noname-a con 0x7f07780b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient(hunting): picked mon.noname-c con 0x7f077813a770 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient(hunting): picked mon.noname-b con 0x7f07781e0170 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07781319e0 con 0x7f07780b28d0 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0778146020 con 0x7f07781e0170 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0778146300 con 0x7f077813a770 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f077e358640 10 monclient(hunting): authenticate will time out at 1247.921407s 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f0777fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f07780b28d0 0x7f0778130bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:49848/0 (socket says 172.21.15.3:49848) 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.667+0000 7f0777fff640 1 -- 172.21.15.3:0/2554672790 learned_addr learned my addr 172.21.15.3:0/2554672790 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:00.653 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter ms_handle_connect 0x7f07781e0170 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter ms_handle_connect 0x7f077813a770 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter ms_handle_connect 0x7f07780b28d0 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.668+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34514389 0 0) 0x7f0778146020 con 0x7f07781e0170 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0764004eb0 con 0x7f07781e0170 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308282547 0 0) 0x7f0778146300 con 0x7f077813a770 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.654 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0778146020 con 0x7f077813a770 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743446639 0 0) 0x7f07781319e0 con 0x7f07780b28d0 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.669+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0778146300 con 0x7f07780b28d0 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 424262147 0 0) 0x7f0764004eb0 con 0x7f07781e0170 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07781319e0 con 0x7f07781e0170 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0770001b90 con 0x7f07781e0170 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.655 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 489786128 0 0) 0x7f0778146020 con 0x7f077813a770 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0764004eb0 con 0x7f077813a770 2024-03-29T11:22:00.656 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f076c001b00 con 0x7f077813a770 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.657 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1945888424 0 0) 0x7f0778146300 con 0x7f07780b28d0 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0778146020 con 0x7f07780b28d0 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f07680019b0 con 0x7f07780b28d0 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.670+0000 7f0774ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.658 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1879967179 0 0) 0x7f07781319e0 con 0x7f07781e0170 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 >> v1:172.21.15.3:6790/0 conn(0x7f077813a770 legacy=0x7f07781312d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 >> v1:172.21.15.3:6789/0 conn(0x7f07780b28d0 legacy=0x7f0778130bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 monclient: found mon.b 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077813f560 con 0x7f07781e0170 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 5 monclient: authenticate success, global_id 4501 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 10 monclient: _renew_subs 2024-03-29T11:22:00.659 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f077813ec60 con 0x7f07781e0170 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 10 monclient: _renew_subs 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0778144e90 con 0x7f07781e0170 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f077e358640 1 librados: init done 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0770002080 con 0x7f07781e0170 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f0770003b10 con 0x7f07781e0170 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f07767fc640 4 set_mon_vals no callback set 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.671+0000 7f0774ff9640 10 monclient: dump: 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:00.660 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.674+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f0770003c70 con 0x7f07781e0170 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.674+0000 7f077e358640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.674+0000 7f077e358640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.674+0000 7f077e358640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:22:00.661 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.674+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0758005080 con 0x7f07781e0170 2024-03-29T11:22:00.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0777fff640 10 monclient: get_auth_request con 0x7f076406e120 auth_method 0 2024-03-29T11:22:00.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f0770083ac0 con 0x7f07781e0170 2024-03-29T11:22:00.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0774ff9640 10 client.4501.objecter ms_dispatch 0x7f0778000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:00.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0774ff9640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:00.662 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0774ff9640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:00.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.675+0000 7f0774ff9640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-29T11:22:00.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.676+0000 7f0774ff9640 10 client.4501.objecter ms_handle_connect 0x7f076406e120 2024-03-29T11:22:00.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.681+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f077004f3a0 con 0x7f07781e0170 2024-03-29T11:22:00.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.681+0000 7f0774ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:00.663 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:00.681+0000 7f0774ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.104+0000 7f077e358640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.104+0000 7f077e358640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.104+0000 7f077e358640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.104+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0758005330 con 0x7f07781e0170 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.105+0000 7f0774ff9640 1 -- 172.21.15.3:0/2554672790 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f07700512d0 con 0x7f07781e0170 2024-03-29T11:22:01.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.105+0000 7f0774ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:01.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.105+0000 7f0774ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:01.082 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:01.082 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:01.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 10 librados: watch_flush enter 2024-03-29T11:22:01.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 10 librados: watch_flush exit 2024-03-29T11:22:01.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-29T11:22:01.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 10 client.4501.objecter successfully canceled tick 2024-03-29T11:22:01.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 >> v1:172.21.15.3:6816/844689376 conn(0x7f076406e120 legacy=0x7f07640705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 10 monclient: shutdown 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.109+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 >> v1:172.21.15.32:6789/0 conn(0x7f07781e0170 legacy=0x7f07781e2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.110+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 shutdown_connections 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.110+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 >> 172.21.15.3:0/2554672790 conn(0x7f0778066a00 msgr2=0x7f07780b69e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.110+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 shutdown_connections 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.110+0000 7f077e358640 1 -- 172.21.15.3:0/2554672790 wait complete. 2024-03-29T11:22:01.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:01.110+0000 7f077e358640 1 librados: shutdown 2024-03-29T11:22:01.103 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:07.909 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:08.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.091+0000 7fc5c8f15640 10 monclient: get_monmap_and_config 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.091+0000 7fc5c8f15640 10 monclient: build_initial_monmap 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.091+0000 7fc5c8f15640 10 monclient: monmap: 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:08.092275+0000 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:08.092275+0000 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:08.068 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:08.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.094+0000 7fc5c8f15640 1 Processor -- start 2024-03-29T11:22:08.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.094+0000 7fc5c8f15640 1 -- start start 2024-03-29T11:22:08.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.094+0000 7fc5c8f15640 10 monclient: init 2024-03-29T11:22:08.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:08.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:08.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c40a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:08.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c40aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:08.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c4146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40aed20 con 0x7fc5c40aab00 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c40aa8b0 con 0x7fc5c40a98a0 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c4146a60 con 0x7fc5c4146f80 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.095+0000 7fc5c8f15640 10 monclient(hunting): authenticate will time out at 1255.349852s 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.096+0000 7fc5c2d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fc5c4146f80 0x7fc5c4151cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:47482/0 (socket says 172.21.15.3:47482) 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.096+0000 7fc5c2d9d640 1 -- 172.21.15.3:0/136810763 learned_addr learned my addr 172.21.15.3:0/136810763 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077640937 0 0) 0x7fc5c4146a60 con 0x7fc5c4146f80 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:22:08.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.100+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac003520 con 0x7fc5c4146f80 2024-03-29T11:22:08.081 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.103+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1997678778 0 0) 0x7fc5ac003520 con 0x7fc5c4146f80 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.103+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 >> v1:172.21.15.32:6789/0 conn(0x7fc5c40a98a0 legacy=0x7fc5c40ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.103+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 >> v1:172.21.15.3:6789/0 conn(0x7fc5c40aab00 legacy=0x7fc5c41437e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 10 monclient: _finish_hunting 0 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 1 monclient: found mon.noname-b 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c4143ef0 con 0x7fc5c4146f80 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 10 monclient: _finish_auth 0 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 10 monclient: _check_auth_tickets 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.104+0000 7fc5c159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c8f15640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:22:08.082 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c8f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc5bc001a10 con 0x7fc5c4146f80 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 10 monclient: dump: 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:08.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c8f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:08.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 1 -- 172.21.15.3:0/136810763 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5bc001ca0 con 0x7fc5c4146f80 2024-03-29T11:22:08.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c8f15640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.105+0000 7fc5c8f15640 4 set_mon_vals no callback set 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.106+0000 7fc5c8f15640 10 monclient: shutdown 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.106+0000 7fc5c8f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.106+0000 7fc5c8f15640 1 -- 172.21.15.3:0/136810763 >> v1:172.21.15.3:6790/0 conn(0x7fc5c4146f80 legacy=0x7fc5c4151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.107+0000 7fc5c8f15640 1 -- 172.21.15.3:0/136810763 shutdown_connections 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.107+0000 7fc5c8f15640 1 -- 172.21.15.3:0/136810763 >> 172.21.15.3:0/136810763 conn(0x7fc5c40b3c10 msgr2=0x7fc5c40b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:08.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.109+0000 7fc5c8f15640 1 -- 172.21.15.3:0/136810763 shutdown_connections 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.109+0000 7fc5c8f15640 1 -- 172.21.15.3:0/136810763 wait complete. 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.110+0000 7fc5c8f15640 10 monclient: build_initial_monmap 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.110+0000 7fc5c8f15640 10 monclient: monmap: 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:08.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.110+0000 7fc5c8f15640 1 librados: starting msgr at 2024-03-29T11:22:08.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.110+0000 7fc5c8f15640 1 librados: starting objecter 2024-03-29T11:22:08.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 1 Processor -- start 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 1 -- start start 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 1 librados: setting wanted keys 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 1 librados: calling monclient init 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 10 monclient: init 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.111+0000 7fc5c8f15640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c40a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c40aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c4146f80 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c406e490 con 0x7fc5c4146f80 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c41525f0 con 0x7fc5c40a98a0 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c406fe10 con 0x7fc5c40aab00 2024-03-29T11:22:08.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:08.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.112+0000 7fc5c8f15640 10 monclient(hunting): authenticate will time out at 1255.366156s 2024-03-29T11:22:08.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.114+0000 7fc5c2d9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fc5c4146f80 0x7fc5c4151ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:34357/0 (socket says 172.21.15.3:34357) 2024-03-29T11:22:08.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.114+0000 7fc5c2d9d640 1 -- 172.21.15.3:0/167264973 learned_addr learned my addr 172.21.15.3:0/167264973 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.115+0000 7fc5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c4146f80 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.115+0000 7fc5b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.115+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690129256 0 0) 0x7fc5c406e490 con 0x7fc5c4146f80 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4004eb0 con 0x7fc5c4146f80 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 client.?.objecter ms_handle_connect 0x7fc5c40aab00 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.116+0000 7fc5b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:08.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086910464 0 0) 0x7fc5a4004eb0 con 0x7fc5c4146f80 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c406e490 con 0x7fc5c4146f80 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc5bc0013a0 con 0x7fc5c4146f80 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:08.095 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:08.096 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:08.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:08.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928555987 0 0) 0x7fc5c406fe10 con 0x7fc5c40aab00 2024-03-29T11:22:08.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4004eb0 con 0x7fc5c40aab00 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3623060405 0 0) 0x7fc5c406e490 con 0x7fc5c4146f80 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 >> v1:172.21.15.3:6790/0 conn(0x7fc5c40aab00 legacy=0x7fc5c41517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.117+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 >> v1:172.21.15.32:6789/0 conn(0x7fc5c40a98a0 legacy=0x7fc5c41441b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:08.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 1 monclient: found mon.a 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c406e920 con 0x7fc5c4146f80 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5bc005240 con 0x7fc5c4146f80 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c0d99640 4 set_mon_vals no callback set 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.118+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fc5bc005650 con 0x7fc5c4146f80 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5b2ffd640 10 monclient: dump: 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:08.100 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 10 monclient: _renew_subs 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c41461f0 con 0x7fc5c4146f80 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 10 monclient: _renew_subs 2024-03-29T11:22:08.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c4144aa0 con 0x7fc5c4146f80 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.119+0000 7fc5c8f15640 1 librados: init done 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.122+0000 7fc5c8f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.122+0000 7fc5c8f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.122+0000 7fc5c8f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:08.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.122+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c406a400 con 0x7fc5c4146f80 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.125+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fc5bc001e40 con 0x7fc5c4146f80 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.126+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fc5bc085050 con 0x7fc5c4146f80 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.126+0000 7fc5b2ffd640 10 client.4491.objecter ms_dispatch 0x7fc5c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.126+0000 7fc5b2ffd640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.126+0000 7fc5b2ffd640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.126+0000 7fc5b2ffd640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-29T11:22:08.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.127+0000 7fc5c259c640 10 monclient: get_auth_request con 0x7fc5a406dde0 auth_method 0 2024-03-29T11:22:08.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.127+0000 7fc5b2ffd640 10 client.4491.objecter ms_handle_connect 0x7fc5a406dde0 2024-03-29T11:22:08.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.131+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5bc050930 con 0x7fc5c4146f80 2024-03-29T11:22:08.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.131+0000 7fc5b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:08.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.131+0000 7fc5b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:08.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.477+0000 7fc5c8f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.478+0000 7fc5c8f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.456 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:08.456 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.478+0000 7fc5c8f15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.478+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5c4123dc0 con 0x7fc5c4146f80 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.480+0000 7fc5b2ffd640 1 -- 172.21.15.3:0/167264973 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fc5bc052860 con 0x7fc5c4146f80 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.480+0000 7fc5b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.480+0000 7fc5b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.484+0000 7fc5c8f15640 10 librados: watch_flush enter 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.484+0000 7fc5c8f15640 10 librados: watch_flush exit 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.484+0000 7fc5c8f15640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.484+0000 7fc5c8f15640 10 client.4491.objecter successfully canceled tick 2024-03-29T11:22:08.475 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 >> v1:172.21.15.3:6816/844689376 conn(0x7fc5a406dde0 legacy=0x7fc5a40702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 10 monclient: shutdown 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 >> v1:172.21.15.3:6789/0 conn(0x7fc5c4146f80 legacy=0x7fc5c4151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 shutdown_connections 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 >> 172.21.15.3:0/167264973 conn(0x7fc5c40b3c10 msgr2=0x7fc5c41435b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 shutdown_connections 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 -- 172.21.15.3:0/167264973 wait complete. 2024-03-29T11:22:08.476 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:08.485+0000 7fc5c8f15640 1 librados: shutdown 2024-03-29T11:22:08.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:15.296 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.449+0000 7faa03eba640 10 monclient: get_monmap_and_config 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.449+0000 7faa03eba640 10 monclient: build_initial_monmap 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.449+0000 7faa03eba640 10 monclient: monmap: 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:15.449989+0000 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:15.449989+0000 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:15.425 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.450+0000 7faa03eba640 1 Processor -- start 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.450+0000 7faa03eba640 1 -- start start 2024-03-29T11:22:15.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.450+0000 7faa03eba640 10 monclient: init 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc0b4f20 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc0a9a90 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0aabd0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0aaeb0 con 0x7fa9fc0b4b40 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0ab190 con 0x7fa9fc0b4f20 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa03eba640 10 monclient(hunting): authenticate will time out at 1262.705386s 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa02457640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa9fc0a9a90 0x7fa9fc145d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:36168/0 (socket says 172.21.15.3:36168) 2024-03-29T11:22:15.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa02457640 1 -- 172.21.15.3:0/3532044123 learned_addr learned my addr 172.21.15.3:0/3532044123 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa01455640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fa9fc0b4f20 0x7fa9fc1455f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55530/0 (socket says 172.21.15.3:55530) 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365742606 0 0) 0x7fa9fc0ab190 con 0x7fa9fc0b4f20 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8003520 con 0x7fa9fc0b4f20 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2625606329 0 0) 0x7fa9fc0aabd0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.451+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc0ab190 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.452+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423396742 0 0) 0x7fa9fc0ab190 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.452+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 >> v1:172.21.15.3:6790/0 conn(0x7fa9fc0b4f20 legacy=0x7fa9fc1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.452+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 >> v1:172.21.15.32:6789/0 conn(0x7fa9fc0b4b40 legacy=0x7fa9fc13be50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: _finish_hunting 0 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 1 monclient: found mon.noname-a 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0b45a0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: _finish_auth 0 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: _check_auth_tickets 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa9ec0015a0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa00c54640 10 monclient: dump: 2024-03-29T11:22:15.432 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa03eba640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.453+0000 7faa03eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9ec0017f0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:15.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 1 -- 172.21.15.3:0/3532044123 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa9ec001b80 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa00c54640 10 monclient: dump: 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa03eba640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.454+0000 7faa03eba640 4 set_mon_vals no callback set 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 10 monclient: shutdown 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 1 -- 172.21.15.3:0/3532044123 >> v1:172.21.15.3:6789/0 conn(0x7fa9fc0a9a90 legacy=0x7fa9fc145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 1 -- 172.21.15.3:0/3532044123 shutdown_connections 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 1 -- 172.21.15.3:0/3532044123 >> 172.21.15.3:0/3532044123 conn(0x7fa9fc066a00 msgr2=0x7fa9fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 1 -- 172.21.15.3:0/3532044123 shutdown_connections 2024-03-29T11:22:15.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.455+0000 7faa03eba640 1 -- 172.21.15.3:0/3532044123 wait complete. 2024-03-29T11:22:15.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.456+0000 7faa03eba640 10 monclient: build_initial_monmap 2024-03-29T11:22:15.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.456+0000 7faa03eba640 10 monclient: monmap: 2024-03-29T11:22:15.439 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.456+0000 7faa03eba640 1 librados: starting msgr at 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.456+0000 7faa03eba640 1 librados: starting objecter 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 1 Processor -- start 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 1 -- start start 2024-03-29T11:22:15.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 1 librados: setting wanted keys 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 1 librados: calling monclient init 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 10 monclient: init 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc0a9a90 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.457+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc143b00 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc12db80 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa02457640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fa9fc143b00 0x7fa9fc12d470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55650/0 (socket says 172.21.15.3:55650) 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa02457640 1 -- 172.21.15.3:0/2742876322 learned_addr learned my addr 172.21.15.3:0/2742876322 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc12ec70 con 0x7fa9fc0b4b40 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc12ef50 con 0x7fa9fc143b00 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:15.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7faa03eba640 10 monclient(hunting): authenticate will time out at 1262.712223s 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7fa9f27fc640 10 client.?.objecter ms_handle_connect 0x7fa9fc143b00 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7fa9f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7fa9f27fc640 10 client.?.objecter ms_handle_connect 0x7fa9fc0a9a90 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.458+0000 7fa9f27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867697646 0 0) 0x7fa9fc12ef50 con 0x7fa9fc143b00 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e0004eb0 con 0x7fa9fc143b00 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197944195 0 0) 0x7fa9fc12db80 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:15.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc12ef50 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1579571017 0 0) 0x7fa9e0004eb0 con 0x7fa9fc143b00 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9fc12db80 con 0x7fa9fc143b00 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa9ec001b40 con 0x7fa9fc143b00 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): dump: 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.443 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 192628750 0 0) 0x7fa9fc12ef50 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e0004eb0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa9f4001790 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.459+0000 7fa9f27fc640 10 monclient(hunting): dump: 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.444 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.460+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2306709760 0 0) 0x7fa9e0004eb0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.460+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 >> v1:172.21.15.3:6790/0 conn(0x7fa9fc143b00 legacy=0x7fa9fc12d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.460+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 >> v1:172.21.15.32:6789/0 conn(0x7fa9fc0b4b40 legacy=0x7fa9fc12cd60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 1 monclient: found mon.a 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc145cd0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 10 monclient: _finish_auth 0 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7fa9f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 10 monclient: _renew_subs 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc145960 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 10 monclient: _renew_subs 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.461+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc145030 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.462+0000 7faa03eba640 1 librados: init done 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f4001d20 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa9f4003790 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f3fff640 4 set_mon_vals no callback set 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7fa9f27fc640 10 monclient: dump: 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:15.447 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:15.448 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:15.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.463+0000 7faa03eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7faa03eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7faa03eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e4005080 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fa9f40197e0 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fa9f4083840 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 10 client.4497.objecter ms_dispatch 0x7fa9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.464+0000 7fa9f27fc640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.465+0000 7faa01455640 10 monclient: get_auth_request con 0x7fa9e006de80 auth_method 0 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.465+0000 7fa9f27fc640 10 client.4497.objecter ms_handle_connect 0x7fa9e006de80 2024-03-29T11:22:15.449 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.471+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9f404ef50 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.471+0000 7fa9f27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:15.450 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.471+0000 7fa9f27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7faa03eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7faa03eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7faa03eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9e4005330 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7fa9f27fc640 1 -- 172.21.15.3:0/2742876322 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fa9f4051050 con 0x7fa9fc0a9a90 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7fa9f27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.766+0000 7fa9f27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:15.743 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 10 librados: watch_flush enter 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 10 librados: watch_flush exit 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 10 client.4497.objecter successfully canceled tick 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 >> v1:172.21.15.3:6816/844689376 conn(0x7fa9e006de80 legacy=0x7fa9e0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 10 monclient: shutdown 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 >> v1:172.21.15.3:6789/0 conn(0x7fa9fc0a9a90 legacy=0x7fa9fc1433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 shutdown_connections 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 >> 172.21.15.3:0/2742876322 conn(0x7fa9fc066a00 msgr2=0x7fa9fc0b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 shutdown_connections 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 -- 172.21.15.3:0/2742876322 wait complete. 2024-03-29T11:22:15.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:15.769+0000 7faa03eba640 1 librados: shutdown 2024-03-29T11:22:15.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:22.566 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.745+0000 7f3a32dcf640 10 monclient: get_monmap_and_config 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.745+0000 7f3a32dcf640 10 monclient: build_initial_monmap 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.745+0000 7f3a32dcf640 10 monclient: monmap: 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:22.745911+0000 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:22.745911+0000 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:22.721 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.722 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:22.722 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:22.722 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:22.722 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.746+0000 7f3a32dcf640 1 Processor -- start 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.746+0000 7f3a32dcf640 1 -- start start 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.746+0000 7f3a32dcf640 10 monclient: init 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c0b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c0b4f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c0a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:22.734 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0aabd0 con 0x7f3a2c0b4b40 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0aaeb0 con 0x7f3a2c0b4f20 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c0ab190 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a32dcf640 10 monclient(hunting): authenticate will time out at 1270.001514s 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a3136c640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3a2c0a9a90 0x7f3a2c145d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:52814/0 (socket says 172.21.15.3:52814) 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.747+0000 7f3a3136c640 1 -- 172.21.15.3:0/320325994 learned_addr learned my addr 172.21.15.3:0/320325994 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786665864 0 0) 0x7f3a2c0ab190 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:22:22.735 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a14003520 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018554884 0 0) 0x7f3a2c0aaeb0 con 0x7f3a2c0b4f20 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c0ab190 con 0x7f3a2c0b4f20 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.748+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402865340 0 0) 0x7f3a2c0aabd0 con 0x7f3a2c0b4b40 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c0aaeb0 con 0x7f3a2c0b4b40 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940338856 0 0) 0x7f3a14003520 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.736 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 >> v1:172.21.15.32:6789/0 conn(0x7f3a2c0b4f20 legacy=0x7f3a2c1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 >> v1:172.21.15.3:6789/0 conn(0x7f3a2c0b4b40 legacy=0x7f3a2c13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c0b45a0 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a32dcf640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a32dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a24001a10 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:22.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a2b7fe640 10 monclient: dump: 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.749+0000 7f3a32dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a24001c30 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a32dcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a32dcf640 4 set_mon_vals no callback set 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/320325994 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a24001fc0 con 0x7f3a2c0a9a90 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a2b7fe640 10 monclient: dump: 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.739 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.750+0000 7f3a32dcf640 10 monclient: shutdown 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 1 -- 172.21.15.3:0/320325994 >> v1:172.21.15.3:6790/0 conn(0x7f3a2c0a9a90 legacy=0x7f3a2c145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 1 -- 172.21.15.3:0/320325994 shutdown_connections 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 1 -- 172.21.15.3:0/320325994 >> 172.21.15.3:0/320325994 conn(0x7f3a2c066a00 msgr2=0x7f3a2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 1 -- 172.21.15.3:0/320325994 shutdown_connections 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.751+0000 7f3a32dcf640 1 -- 172.21.15.3:0/320325994 wait complete. 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 10 monclient: build_initial_monmap 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 10 monclient: monmap: 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:22.740 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 librados: starting msgr at 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 librados: starting objecter 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 Processor -- start 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 -- start start 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 librados: setting wanted keys 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 1 librados: calling monclient init 2024-03-29T11:22:22.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.752+0000 7f3a32dcf640 10 monclient: init 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f3a2c0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f3a2c06f5e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f3a2c1e40c0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c06ec00 con 0x7f3a2c1e40c0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c06eee0 con 0x7f3a2c0b4b40 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a2c132900 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a32dcf640 10 monclient(hunting): authenticate will time out at 1270.007697s 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a30b6b640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3a2c0b4b40 0x7f3a2c13ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:55388/0 (socket says 172.21.15.3:55388) 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.753+0000 7f3a30b6b640 1 -- 172.21.15.3:0/1954404504 learned_addr learned my addr 172.21.15.3:0/1954404504 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 client.?.objecter ms_handle_connect 0x7f3a2c06f5e0 2024-03-29T11:22:22.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288749515 0 0) 0x7f3a2c132900 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a18004eb0 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 client.?.objecter ms_handle_connect 0x7f3a2c1e40c0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1216778051 0 0) 0x7f3a18004eb0 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a2c132900 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a20001970 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.754+0000 7f3a297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:22.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): dump: 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 client.?.objecter ms_handle_connect 0x7f3a2c0b4b40 2024-03-29T11:22:22.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408951190 0 0) 0x7f3a2c06ec00 con 0x7f3a2c1e40c0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a18004eb0 con 0x7f3a2c1e40c0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 355006136 0 0) 0x7f3a2c132900 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 >> v1:172.21.15.32:6789/0 conn(0x7f3a2c0b4b40 legacy=0x7f3a2c13ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 >> v1:172.21.15.3:6789/0 conn(0x7f3a2c1e40c0 legacy=0x7f3a2c1e64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 monclient: found mon.c 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c1463b0 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:22.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 10 monclient: _renew_subs 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a2c145390 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 10 monclient: _renew_subs 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a2c06e060 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a32dcf640 1 librados: init done 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a20001b90 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:22.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a20003a60 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a297fa640 10 monclient: dump: 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:22.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.755+0000 7f3a2affd640 4 set_mon_vals no callback set 2024-03-29T11:22:22.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.757+0000 7f3a32dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.757+0000 7f3a32dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.757+0000 7f3a32dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:22.748 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.757+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c005080 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.758+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f3a20019a30 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f3a20083960 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 10 client.4532.objecter ms_dispatch 0x7f3a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a30b6b640 10 monclient: get_auth_request con 0x7f3a1806dcb0 auth_method 0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.759+0000 7f3a297fa640 10 client.4532.objecter ms_handle_connect 0x7f3a1806dcb0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.768+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a2004f2a0 con 0x7f3a2c06f5e0 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.768+0000 7f3a297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:22.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:22.768+0000 7f3a297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:23.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.123+0000 7f3a32dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.123+0000 7f3a32dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.123+0000 7f3a32dcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:23.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.123+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a0c005330 con 0x7f3a2c06f5e0 2024-03-29T11:22:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.124+0000 7f3a297fa640 1 -- 172.21.15.3:0/1954404504 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f3a200511d0 con 0x7f3a2c06f5e0 2024-03-29T11:22:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.124+0000 7f3a297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:23.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.124+0000 7f3a297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:23.101 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:23.101 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:23.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 10 librados: watch_flush enter 2024-03-29T11:22:23.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 10 librados: watch_flush exit 2024-03-29T11:22:23.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-29T11:22:23.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 10 client.4532.objecter successfully canceled tick 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 >> v1:172.21.15.3:6816/844689376 conn(0x7f3a1806dcb0 legacy=0x7f3a18070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 10 monclient: shutdown 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 >> v1:172.21.15.3:6790/0 conn(0x7f3a2c06f5e0 legacy=0x7f3a2c145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 shutdown_connections 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.127+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 >> 172.21.15.3:0/1954404504 conn(0x7f3a2c066a00 msgr2=0x7f3a2c1357e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.128+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 shutdown_connections 2024-03-29T11:22:23.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.128+0000 7f3a32dcf640 1 -- 172.21.15.3:0/1954404504 wait complete. 2024-03-29T11:22:23.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:23.128+0000 7f3a32dcf640 1 librados: shutdown 2024-03-29T11:22:23.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:29.924 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.092+0000 7f4ea9c38640 10 monclient: get_monmap_and_config 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.092+0000 7f4ea9c38640 10 monclient: build_initial_monmap 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.092+0000 7f4ea9c38640 10 monclient: monmap: 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:30.093666+0000 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:30.093666+0000 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:30.069 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:30.070 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:30.070 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.093+0000 7f4ea9c38640 1 Processor -- start 2024-03-29T11:22:30.070 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.093+0000 7f4ea9c38640 1 -- start start 2024-03-29T11:22:30.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.093+0000 7f4ea9c38640 10 monclient: init 2024-03-29T11:22:30.071 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea40af800 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea40b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea4145cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea8c36640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ea40af800 0x7f4ea41455a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:48012/0 (socket says 172.21.15.3:48012) 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea8c36640 1 -- 172.21.15.3:0/922372593 learned_addr learned my addr 172.21.15.3:0/922372593 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea40b0a00 con 0x7f4ea4145cb0 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea40b17b0 con 0x7f4ea40af800 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea40aed60 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.094+0000 7f4ea9c38640 10 monclient(hunting): authenticate will time out at 1277.348859s 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 731618992 0 0) 0x7f4ea40b17b0 con 0x7f4ea40af800 2024-03-29T11:22:30.072 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9400bd30 con 0x7f4ea40af800 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087184174 0 0) 0x7f4e9400bd30 con 0x7f4ea40af800 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 >> v1:172.21.15.3:6790/0 conn(0x7f4ea40b1cd0 legacy=0x7f4ea413f1b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:30.073 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.095+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 >> v1:172.21.15.3:6789/0 conn(0x7f4ea4145cb0 legacy=0x7f4ea414d8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea40aeef0 con 0x7f4ea40af800 2024-03-29T11:22:30.076 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.098+0000 7f4ea37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4e98001690 con 0x7f4ea40af800 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea9c38640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea9c38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 10 monclient: dump: 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.077 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.099+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e98001930 con 0x7f4ea40af800 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea9c38640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea9c38640 4 set_mon_vals no callback set 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea37fe640 1 -- 172.21.15.3:0/922372593 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4e98001cc0 con 0x7f4ea40af800 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.100+0000 7f4ea37fe640 10 monclient: dump: 2024-03-29T11:22:30.078 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:30.079 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 10 monclient: shutdown 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 >> v1:172.21.15.32:6789/0 conn(0x7f4ea40af800 legacy=0x7f4ea41455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 shutdown_connections 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 >> 172.21.15.3:0/922372593 conn(0x7f4ea4066a00 msgr2=0x7f4ea4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 shutdown_connections 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.101+0000 7f4ea9c38640 1 -- 172.21.15.3:0/922372593 wait complete. 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.102+0000 7f4ea9c38640 10 monclient: build_initial_monmap 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.102+0000 7f4ea9c38640 10 monclient: monmap: 2024-03-29T11:22:30.083 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 librados: starting msgr at 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 librados: starting objecter 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 Processor -- start 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 -- start start 2024-03-29T11:22:30.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 librados: setting wanted keys 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 1 librados: calling monclient init 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.103+0000 7f4ea9c38640 10 monclient: init 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.104+0000 7f4ea9c38640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.104+0000 7f4ea9c38640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.104+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea40af800 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.104+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea40b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea3fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f4ea40b1cd0 0x7f4ea4136e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:55912/0 (socket says 172.21.15.3:55912) 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea4141390 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea3fff640 1 -- 172.21.15.3:0/1110647084 learned_addr learned my addr 172.21.15.3:0/1110647084 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea4141940 con 0x7f4ea40af800 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 client.?.objecter ms_handle_connect 0x7f4ea40b1cd0 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea4141dd0 con 0x7f4ea4141390 2024-03-29T11:22:30.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea413fea0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea9c38640 10 monclient(hunting): authenticate will time out at 1277.359628s 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684978895 0 0) 0x7f4ea413fea0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.105+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88004eb0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2611269047 0 0) 0x7f4e88004eb0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e88005300 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4e90001830 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 10 monclient(hunting): dump: 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:30.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.106+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439320436 0 0) 0x7f4e88005300 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.107+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 >> v1:172.21.15.32:6789/0 conn(0x7f4ea4141390 legacy=0x7f4ea4138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.107+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 >> v1:172.21.15.3:6789/0 conn(0x7f4ea40af800 legacy=0x7f4ea4140b10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:30.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 1 monclient: found mon.c 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4137530 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 10 monclient: _renew_subs 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea41377e0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 10 monclient: _renew_subs 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:30.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea41396f0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea9c38640 1 librados: init done 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e90001e00 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f4e90001830 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.108+0000 7f4ea17fa640 10 monclient: dump: 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:30.093 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:30.094 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.109+0000 7f4ea2ffd640 4 set_mon_vals no callback set 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.110+0000 7f4ea9c38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.110+0000 7f4ea9c38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.110+0000 7f4ea9c38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.111+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e84005080 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.111+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f4e9001a8d0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f4e900846d0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 10 client.4535.objecter ms_dispatch 0x7f4ea4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:30.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-29T11:22:30.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea8c36640 10 monclient: get_auth_request con 0x7f4e8806e210 auth_method 0 2024-03-29T11:22:30.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.112+0000 7f4ea17fa640 10 client.4535.objecter ms_handle_connect 0x7f4e8806e210 2024-03-29T11:22:30.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.120+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e9004fde0 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.120+0000 7f4ea17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:30.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.120+0000 7f4ea17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:30.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.428+0000 7f4ea9c38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.428+0000 7f4ea9c38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.428+0000 7f4ea9c38640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:30.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.428+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e84005330 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.429+0000 7f4ea17fa640 1 -- 172.21.15.3:0/1110647084 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f4e90001690 con 0x7f4ea40b1cd0 2024-03-29T11:22:30.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.429+0000 7f4ea17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:30.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.429+0000 7f4ea17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:30.405 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:30.406 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.432+0000 7f4ea9c38640 10 librados: watch_flush enter 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.432+0000 7f4ea9c38640 10 librados: watch_flush exit 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.432+0000 7f4ea9c38640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.432+0000 7f4ea9c38640 10 client.4535.objecter successfully canceled tick 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.432+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 >> v1:172.21.15.3:6816/844689376 conn(0x7f4e8806e210 legacy=0x7f4e880706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 10 monclient: shutdown 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:30.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 >> v1:172.21.15.3:6790/0 conn(0x7f4ea40b1cd0 legacy=0x7f4ea4136e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:30.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 shutdown_connections 2024-03-29T11:22:30.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 >> 172.21.15.3:0/1110647084 conn(0x7f4ea4066a00 msgr2=0x7f4ea40b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:30.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.433+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 shutdown_connections 2024-03-29T11:22:30.410 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.434+0000 7f4ea9c38640 1 -- 172.21.15.3:0/1110647084 wait complete. 2024-03-29T11:22:30.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:30.434+0000 7f4ea9c38640 1 librados: shutdown 2024-03-29T11:22:30.424 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:37.229 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.386+0000 7f7dc985c640 10 monclient: get_monmap_and_config 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.386+0000 7f7dc985c640 10 monclient: build_initial_monmap 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.386+0000 7f7dc985c640 10 monclient: monmap: 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:37.387378+0000 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:37.387378+0000 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.363 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.387+0000 7f7dc985c640 1 Processor -- start 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.387+0000 7f7dc985c640 1 -- start start 2024-03-29T11:22:37.364 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.387+0000 7f7dc985c640 10 monclient: init 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc40b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc40b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc40b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40aab60 con 0x7f7dc40b3560 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40aae40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc40ab120 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.388+0000 7f7dc985c640 10 monclient(hunting): authenticate will time out at 1284.642796s 2024-03-29T11:22:37.365 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc2ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f7dc40b28d0 0x7f7dc413bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:33658/0 (socket says 172.21.15.3:33658) 2024-03-29T11:22:37.366 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc2ffd640 1 -- 172.21.15.3:0/3393022874 learned_addr learned my addr 172.21.15.3:0/3393022874 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595800413 0 0) 0x7f7dc40aae40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db0003520 con 0x7f7dc40b28d0 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2246453185 0 0) 0x7f7dc40ab120 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.368 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.389+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc40aae40 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.390+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494232556 0 0) 0x7f7dc40aae40 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.390+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 >> v1:172.21.15.32:6789/0 conn(0x7f7dc40b28d0 legacy=0x7f7dc413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.390+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 >> v1:172.21.15.3:6789/0 conn(0x7f7dc40b3560 legacy=0x7f7dc41428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.390+0000 7f7dc1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.390+0000 7f7dc1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc413c380 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc985c640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc985c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:37.369 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7db4001670 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: dump: 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc985c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:37.370 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7db40017d0 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 1 -- 172.21.15.3:0/3393022874 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7db4001c20 con 0x7f7dc40b2cb0 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.391+0000 7f7dc1ffb640 10 monclient: dump: 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:37.371 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.392+0000 7f7dc985c640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:37.377 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.392+0000 7f7dc985c640 4 set_mon_vals no callback set 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.392+0000 7f7dc985c640 10 monclient: shutdown 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.392+0000 7f7dc985c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.392+0000 7f7dc985c640 1 -- 172.21.15.3:0/3393022874 >> v1:172.21.15.3:6790/0 conn(0x7f7dc40b2cb0 legacy=0x7f7dc4142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.393+0000 7f7dc985c640 1 -- 172.21.15.3:0/3393022874 shutdown_connections 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.393+0000 7f7dc985c640 1 -- 172.21.15.3:0/3393022874 >> 172.21.15.3:0/3393022874 conn(0x7f7dc4066a00 msgr2=0x7f7dc4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.393+0000 7f7dc985c640 1 -- 172.21.15.3:0/3393022874 shutdown_connections 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.393+0000 7f7dc985c640 1 -- 172.21.15.3:0/3393022874 wait complete. 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 10 monclient: build_initial_monmap 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 10 monclient: monmap: 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:37.378 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 librados: starting msgr at 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 librados: starting objecter 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 Processor -- start 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 -- start start 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 librados: setting wanted keys 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 1 librados: calling monclient init 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.394+0000 7f7dc985c640 10 monclient: init 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:37.379 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-c con 0x7f7dc40b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-a con 0x7f7dc413e760 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 10 monclient(hunting): picked mon.noname-b con 0x7f7dc41db360 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.395+0000 7f7dc985c640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc41da750 con 0x7f7dc413e760 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc985c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc41d3cb0 con 0x7f7dc41db360 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc2ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f7dc40b28d0 0x7f7dc4144e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:59642/0 (socket says 172.21.15.3:59642) 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc2ffd640 1 -- 172.21.15.3:0/1887734874 learned_addr learned my addr 172.21.15.3:0/1887734874 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc41d3e40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc985c640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dc985c640 10 monclient(hunting): authenticate will time out at 1284.650281s 2024-03-29T11:22:37.380 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 client.?.objecter ms_handle_connect 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482775113 0 0) 0x7f7dc41d3e40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.396+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da4004eb0 con 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728550897 0 0) 0x7f7da4004eb0 con 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7dc41d3e40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7dbc001d10 con 0x7f7dc40b28d0 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 10 monclient(hunting): dump: 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:37.381 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2473652598 0 0) 0x7f7dc41d3e40 con 0x7f7dc40b28d0 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 >> v1:172.21.15.32:6789/0 conn(0x7f7dc41db360 legacy=0x7f7dc41d4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.382 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.397+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 >> v1:172.21.15.3:6789/0 conn(0x7f7dc413e760 legacy=0x7f7dc41dac50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 1 monclient: found mon.c 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc413dc20 con 0x7f7dc40b28d0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 10 monclient: _finish_auth 0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 10 monclient: _renew_subs 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc413de70 con 0x7f7dc40b28d0 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:37.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 10 monclient: _renew_subs 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc41d4b70 con 0x7f7dc40b28d0 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc985c640 1 librados: init done 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7dbc004b60 con 0x7f7dc40b28d0 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.399+0000 7f7dc17fa640 4 set_mon_vals no callback set 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f7dbc004fb0 con 0x7f7dc40b28d0 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dbbfff640 10 monclient: dump: 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:37.384 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dc985c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dc985c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dc985c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.400+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da8005080 con 0x7f7dc40b28d0 2024-03-29T11:22:37.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.401+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f7dbc005110 con 0x7f7dc40b28d0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f7dbc084ef0 con 0x7f7dc40b28d0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dc27fc640 10 monclient: get_auth_request con 0x7f7da406dbc0 auth_method 0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 10 client.4541.objecter ms_dispatch 0x7f7dc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:22:37.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.402+0000 7f7dbbfff640 10 client.4541.objecter ms_handle_connect 0x7f7da406dbc0 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.410+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7dbc050830 con 0x7f7dc40b28d0 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.410+0000 7f7dbbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:37.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.410+0000 7f7dbbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.726+0000 7f7dc985c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.726+0000 7f7dc985c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.726+0000 7f7dc985c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.726+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7da8005330 con 0x7f7dc40b28d0 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.727+0000 7f7dbbfff640 1 -- 172.21.15.3:0/1887734874 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f7dbc052760 con 0x7f7dc40b28d0 2024-03-29T11:22:37.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.727+0000 7f7dbbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:37.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.727+0000 7f7dbbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:37.704 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:37.704 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:37.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.730+0000 7f7dc985c640 10 librados: watch_flush enter 2024-03-29T11:22:37.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.730+0000 7f7dc985c640 10 librados: watch_flush exit 2024-03-29T11:22:37.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.730+0000 7f7dc985c640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:22:37.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.730+0000 7f7dc985c640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.730+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 >> v1:172.21.15.3:6816/844689376 conn(0x7f7da406dbc0 legacy=0x7f7da4070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 10 monclient: shutdown 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 >> v1:172.21.15.3:6790/0 conn(0x7f7dc40b28d0 legacy=0x7f7dc4144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 shutdown_connections 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 >> 172.21.15.3:0/1887734874 conn(0x7f7dc4066a00 msgr2=0x7f7dc413fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 shutdown_connections 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 -- 172.21.15.3:0/1887734874 wait complete. 2024-03-29T11:22:37.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:37.731+0000 7f7dc985c640 1 librados: shutdown 2024-03-29T11:22:37.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:44.528 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:44.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.696+0000 7fee53412640 10 monclient: get_monmap_and_config 2024-03-29T11:22:44.674 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.696+0000 7fee53412640 10 monclient: build_initial_monmap 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.696+0000 7fee53412640 10 monclient: monmap: 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:44.698362+0000 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:44.698362+0000 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:44.675 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:44.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.697+0000 7fee53412640 1 Processor -- start 2024-03-29T11:22:44.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.697+0000 7fee53412640 1 -- start start 2024-03-29T11:22:44.676 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.697+0000 7fee53412640 10 monclient: init 2024-03-29T11:22:44.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:44.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:44.677 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient(hunting): picked mon.noname-b con 0x7fee44006e30 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient(hunting): picked mon.noname-a con 0x7fee44035e00 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient(hunting): picked mon.noname-c con 0x7fee440c9670 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee44035970 con 0x7fee44035e00 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee44036280 con 0x7fee440c9670 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee44035640 con 0x7fee44006e30 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:44.679 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.699+0000 7fee53412640 10 monclient(hunting): authenticate will time out at 1291.954782s 2024-03-29T11:22:44.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee52410640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fee44006e30 0x7fee440cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:44136/0 (socket says 172.21.15.3:44136) 2024-03-29T11:22:44.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee52410640 1 -- 172.21.15.3:0/2853992001 learned_addr learned my addr 172.21.15.3:0/2853992001 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:44.681 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179623848 0 0) 0x7fee44036280 con 0x7fee440c9670 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.703+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee38003520 con 0x7fee440c9670 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338575788 0 0) 0x7fee44035970 con 0x7fee44035e00 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44036280 con 0x7fee44035e00 2024-03-29T11:22:44.682 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283983842 0 0) 0x7fee44035640 con 0x7fee44006e30 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44035970 con 0x7fee44006e30 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726036492 0 0) 0x7fee38003520 con 0x7fee440c9670 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 >> v1:172.21.15.3:6790/0 conn(0x7fee44006e30 legacy=0x7fee440cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 >> v1:172.21.15.3:6789/0 conn(0x7fee44035e00 legacy=0x7fee440c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: _finish_hunting 0 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 monclient: found mon.noname-c 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44034e20 con 0x7fee440c9670 2024-03-29T11:22:44.683 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: _finish_auth 0 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: _check_auth_tickets 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee53412640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee53412640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee3c001690 con 0x7fee440c9670 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee5140e640 10 monclient: dump: 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.684 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:44.685 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:44.685 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:44.685 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:44.685 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:44.685 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.704+0000 7fee53412640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.705+0000 7fee5140e640 1 -- 172.21.15.3:0/2853992001 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee3c0017f0 con 0x7fee440c9670 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.705+0000 7fee5140e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.705+0000 7fee53412640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.705+0000 7fee53412640 4 set_mon_vals no callback set 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 10 monclient: shutdown 2024-03-29T11:22:44.686 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:44.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 1 -- 172.21.15.3:0/2853992001 >> v1:172.21.15.32:6789/0 conn(0x7fee440c9670 legacy=0x7fee440d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:44.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 1 -- 172.21.15.3:0/2853992001 shutdown_connections 2024-03-29T11:22:44.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 1 -- 172.21.15.3:0/2853992001 >> 172.21.15.3:0/2853992001 conn(0x7fee44005c00 msgr2=0x7fee44006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:44.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.706+0000 7fee53412640 1 -- 172.21.15.3:0/2853992001 shutdown_connections 2024-03-29T11:22:44.687 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.707+0000 7fee53412640 1 -- 172.21.15.3:0/2853992001 wait complete. 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.708+0000 7fee53412640 10 monclient: build_initial_monmap 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.708+0000 7fee53412640 10 monclient: monmap: 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:44.688 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.709+0000 7fee53412640 1 librados: starting msgr at 2024-03-29T11:22:44.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.709+0000 7fee53412640 1 librados: starting objecter 2024-03-29T11:22:44.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.709+0000 7fee53412640 1 Processor -- start 2024-03-29T11:22:44.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.710+0000 7fee53412640 1 -- start start 2024-03-29T11:22:44.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.710+0000 7fee53412640 1 librados: setting wanted keys 2024-03-29T11:22:44.689 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.710+0000 7fee53412640 1 librados: calling monclient init 2024-03-29T11:22:44.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.710+0000 7fee53412640 10 monclient: init 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient(hunting): picked mon.noname-c con 0x7fee44006e30 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient(hunting): picked mon.noname-b con 0x7fee44035e00 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient(hunting): picked mon.noname-a con 0x7fee440c9670 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4402dcd0 con 0x7fee440c9670 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee44033a50 con 0x7fee44035e00 2024-03-29T11:22:44.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee4402df20 con 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee53412640 10 monclient(hunting): authenticate will time out at 1291.966695s 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee52410640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fee44006e30 0x7fee44030ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:45892/0 (socket says 172.21.15.3:45892) 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.711+0000 7fee52410640 1 -- 172.21.15.3:0/4179952261 learned_addr learned my addr 172.21.15.3:0/4179952261 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 client.?.objecter ms_handle_connect 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871027176 0 0) 0x7fee4402df20 con 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.712+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee30004eb0 con 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668416193 0 0) 0x7fee30004eb0 con 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee4402df20 con 0x7fee44006e30 2024-03-29T11:22:44.694 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee4c04fdd0 con 0x7fee44006e30 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:44.695 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2074717092 0 0) 0x7fee4402df20 con 0x7fee44006e30 2024-03-29T11:22:44.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.715+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 >> v1:172.21.15.32:6789/0 conn(0x7fee44035e00 legacy=0x7fee4402f710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:44.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.716+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 >> v1:172.21.15.3:6789/0 conn(0x7fee440c9670 legacy=0x7fee4416c350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:44.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 1 monclient: found mon.c 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44033530 con 0x7fee44006e30 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.717+0000 7fee42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 10 monclient: _renew_subs 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee44033cc0 con 0x7fee44006e30 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 10 monclient: _renew_subs 2024-03-29T11:22:44.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee44038d70 con 0x7fee44006e30 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.718+0000 7fee53412640 1 librados: init done 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee53412640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee53412640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee53412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee4c04f430 con 0x7fee44006e30 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee44007b90 con 0x7fee44006e30 2024-03-29T11:22:44.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee4c04eee0 con 0x7fee44006e30 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.720+0000 7fee42ffd640 10 monclient: dump: 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:44.700 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:44.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.721+0000 7fee50c0d640 4 set_mon_vals no callback set 2024-03-29T11:22:44.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.723+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fee4c067be0 con 0x7fee44006e30 2024-03-29T11:22:44.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.724+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fee4c0ff6d0 con 0x7fee44006e30 2024-03-29T11:22:44.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.724+0000 7fee42ffd640 10 client.4547.objecter ms_dispatch 0x7fee4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:44.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.724+0000 7fee42ffd640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:44.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.725+0000 7fee42ffd640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:44.703 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.725+0000 7fee42ffd640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:22:44.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.726+0000 7fee51c0f640 10 monclient: get_auth_request con 0x7fee3006db90 auth_method 0 2024-03-29T11:22:44.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.727+0000 7fee42ffd640 10 client.4547.objecter ms_handle_connect 0x7fee3006db90 2024-03-29T11:22:44.707 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.729+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee4c0cafb0 con 0x7fee44006e30 2024-03-29T11:22:44.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.729+0000 7fee42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:44.708 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:44.729+0000 7fee42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:45.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.076+0000 7fee53412640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.076+0000 7fee53412640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.077+0000 7fee53412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:22:45.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.077+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee44005120 con 0x7fee44006e30 2024-03-29T11:22:45.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.081+0000 7fee42ffd640 1 -- 172.21.15.3:0/4179952261 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fee4c0ccee0 con 0x7fee44006e30 2024-03-29T11:22:45.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.081+0000 7fee42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:45.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.081+0000 7fee42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:45.059 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:45.059 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:45.062 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.084+0000 7fee53412640 10 librados: watch_flush enter 2024-03-29T11:22:45.063 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 10 librados: watch_flush exit 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 >> v1:172.21.15.3:6816/844689376 conn(0x7fee3006db90 legacy=0x7fee30070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 10 monclient: shutdown 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 >> v1:172.21.15.3:6790/0 conn(0x7fee44006e30 legacy=0x7fee44030ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 shutdown_connections 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 >> 172.21.15.3:0/4179952261 conn(0x7fee44005c00 msgr2=0x7fee440c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.085+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 shutdown_connections 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.086+0000 7fee53412640 1 -- 172.21.15.3:0/4179952261 wait complete. 2024-03-29T11:22:45.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:45.086+0000 7fee53412640 1 librados: shutdown 2024-03-29T11:22:45.082 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:51.888 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:52.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.057+0000 7f9c85132640 10 monclient: get_monmap_and_config 2024-03-29T11:22:52.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.057+0000 7f9c85132640 10 monclient: build_initial_monmap 2024-03-29T11:22:52.035 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.057+0000 7f9c85132640 10 monclient: monmap: 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:52.059027+0000 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:52.059027+0000 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:52.036 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:52.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.058+0000 7f9c85132640 1 Processor -- start 2024-03-29T11:22:52.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.058+0000 7f9c85132640 1 -- start start 2024-03-29T11:22:52.038 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.058+0000 7f9c85132640 10 monclient: init 2024-03-29T11:22:52.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.059+0000 7f9c85132640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:52.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.059+0000 7f9c85132640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:52.039 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.059+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-c con 0x7f9c800b28d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:52.040 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-b con 0x7f9c800b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-a con 0x7f9c800b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800aab60 con 0x7f9c800b3560 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800aae40 con 0x7f9c800b28d0 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c800ab120 con 0x7f9c800b2cb0 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c85132640 10 monclient(hunting): authenticate will time out at 1299.317704s 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c7ed9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f9c800b28d0 0x7f9c8013bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:51453/0 (socket says 172.21.15.3:51453) 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.062+0000 7f9c7ed9d640 1 -- 172.21.15.3:0/67938791 learned_addr learned my addr 172.21.15.3:0/67938791 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316080684 0 0) 0x7f9c800aae40 con 0x7f9c800b28d0 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.042 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:22:52.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.063+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c68003520 con 0x7f9c800b28d0 2024-03-29T11:22:52.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.064+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1405579995 0 0) 0x7f9c68003520 con 0x7f9c800b28d0 2024-03-29T11:22:52.043 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.064+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 >> v1:172.21.15.3:6790/0 conn(0x7f9c800b2cb0 legacy=0x7f9c80142190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:52.045 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 >> v1:172.21.15.3:6789/0 conn(0x7f9c800b3560 legacy=0x7f9c801428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c8013c380 con 0x7f9c800b28d0 2024-03-29T11:22:52.048 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c85132640 5 monclient: authenticate success, global_id 4519 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c85132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9c78001a10 con 0x7f9c800b28d0 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: dump: 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.049 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c85132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 1 -- 172.21.15.3:0/67938791 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c78001ca0 con 0x7f9c800b28d0 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c7dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c85132640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.067+0000 7f9c85132640 4 set_mon_vals no callback set 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.068+0000 7f9c85132640 10 monclient: shutdown 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.068+0000 7f9c85132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.068+0000 7f9c85132640 1 -- 172.21.15.3:0/67938791 >> v1:172.21.15.32:6789/0 conn(0x7f9c800b28d0 legacy=0x7f9c8013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.069+0000 7f9c85132640 1 -- 172.21.15.3:0/67938791 shutdown_connections 2024-03-29T11:22:52.050 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.069+0000 7f9c85132640 1 -- 172.21.15.3:0/67938791 >> 172.21.15.3:0/67938791 conn(0x7f9c80066a00 msgr2=0x7f9c80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:52.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.071+0000 7f9c85132640 1 -- 172.21.15.3:0/67938791 shutdown_connections 2024-03-29T11:22:52.051 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.071+0000 7f9c85132640 1 -- 172.21.15.3:0/67938791 wait complete. 2024-03-29T11:22:52.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.072+0000 7f9c85132640 10 monclient: build_initial_monmap 2024-03-29T11:22:52.054 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.072+0000 7f9c85132640 10 monclient: monmap: 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.072+0000 7f9c85132640 1 librados: starting msgr at 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.072+0000 7f9c85132640 1 librados: starting objecter 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.073+0000 7f9c85132640 1 Processor -- start 2024-03-29T11:22:52.055 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.073+0000 7f9c85132640 1 -- start start 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.073+0000 7f9c85132640 1 librados: setting wanted keys 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.073+0000 7f9c85132640 1 librados: calling monclient init 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.073+0000 7f9c85132640 10 monclient: init 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-a con 0x7f9c800b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-c con 0x7f9c80132660 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient(hunting): picked mon.noname-b con 0x7f9c8006f7a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8013f6c0 con 0x7f9c800b28d0 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8006efc0 con 0x7f9c8006f7a0 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c8013ea90 con 0x7f9c80132660 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:52.056 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.074+0000 7f9c85132640 10 monclient(hunting): authenticate will time out at 1299.329165s 2024-03-29T11:22:52.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.079+0000 7f9c7ed9d640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f9c800b28d0 0x7f9c80145e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:60665/0 (socket says 172.21.15.3:60665) 2024-03-29T11:22:52.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.079+0000 7f9c7ed9d640 1 -- 172.21.15.3:0/1712810038 learned_addr learned my addr 172.21.15.3:0/1712810038 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:52.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.079+0000 7f9c677fe640 10 client.?.objecter ms_handle_connect 0x7f9c800b28d0 2024-03-29T11:22:52.059 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.079+0000 7f9c677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030873037 0 0) 0x7f9c8013f6c0 con 0x7f9c800b28d0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c6c004eb0 con 0x7f9c800b28d0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 client.?.objecter ms_handle_connect 0x7f9c8006f7a0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3140868969 0 0) 0x7f9c6c004eb0 con 0x7f9c800b28d0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c6c005300 con 0x7f9c800b28d0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9c78002030 con 0x7f9c800b28d0 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:52.060 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): dump: 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.080+0000 7f9c677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.081+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2990775200 0 0) 0x7f9c6c005300 con 0x7f9c800b28d0 2024-03-29T11:22:52.061 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.081+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 >> v1:172.21.15.3:6790/0 conn(0x7f9c80132660 legacy=0x7f9c8013efb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 >> v1:172.21.15.32:6789/0 conn(0x7f9c8006f7a0 legacy=0x7f9c801ddd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 1 monclient: found mon.a 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c80129880 con 0x7f9c800b28d0 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 10 monclient: _renew_subs 2024-03-29T11:22:52.064 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c8013e6b0 con 0x7f9c800b28d0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 10 monclient: _renew_subs 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c801df620 con 0x7f9c800b28d0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.084+0000 7f9c85132640 1 librados: init done 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c78004e10 con 0x7f9c800b28d0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f9c78002030 con 0x7f9c800b28d0 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c677fe640 10 monclient: dump: 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.065 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:52.066 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.086+0000 7f9c7d59a640 4 set_mon_vals no callback set 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f9c7801adf0 con 0x7f9c800b28d0 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f9c78054a00 con 0x7f9c800b28d0 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 10 client.4512.objecter ms_dispatch 0x7f9c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.087+0000 7f9c677fe640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c85132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c85132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c85132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:52.067 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c60005080 con 0x7f9c800b28d0 2024-03-29T11:22:52.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c7e59c640 10 monclient: get_auth_request con 0x7f9c6c06e080 auth_method 0 2024-03-29T11:22:52.068 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.088+0000 7f9c677fe640 10 client.4512.objecter ms_handle_connect 0x7f9c6c06e080 2024-03-29T11:22:52.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.097+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c78050ca0 con 0x7f9c800b28d0 2024-03-29T11:22:52.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.097+0000 7f9c677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:52.075 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.097+0000 7f9c677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:52.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.424+0000 7f9c85132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.424+0000 7f9c85132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.424+0000 7f9c85132640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:52.402 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.424+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c60005330 con 0x7f9c800b28d0 2024-03-29T11:22:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.425+0000 7f9c677fe640 1 -- 172.21.15.3:0/1712810038 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f9c78050ca0 con 0x7f9c800b28d0 2024-03-29T11:22:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.425+0000 7f9c677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:52.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.425+0000 7f9c677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:52.404 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:52.404 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 10 librados: watch_flush enter 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 10 librados: watch_flush exit 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 10 client.4512.objecter successfully canceled tick 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 >> v1:172.21.15.3:6816/844689376 conn(0x7f9c6c06e080 legacy=0x7f9c6c070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 10 monclient: shutdown 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 >> v1:172.21.15.3:6789/0 conn(0x7f9c800b28d0 legacy=0x7f9c80145e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 shutdown_connections 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 >> 172.21.15.3:0/1712810038 conn(0x7f9c80066a00 msgr2=0x7f9c80135690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:52.408 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.429+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 shutdown_connections 2024-03-29T11:22:52.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.430+0000 7f9c85132640 1 -- 172.21.15.3:0/1712810038 wait complete. 2024-03-29T11:22:52.409 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:52.430+0000 7f9c85132640 1 librados: shutdown 2024-03-29T11:22:52.426 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:59.233 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:59.383 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.406+0000 7fee49d3b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.406+0000 7fee49d3b640 10 monclient: build_initial_monmap 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.406+0000 7fee49d3b640 10 monclient: monmap: 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:22:59.407032+0000 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:22:59.407032+0000 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:22:59.384 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:22:59.385 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.385 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.408+0000 7fee49d3b640 1 Processor -- start 2024-03-29T11:22:59.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.408+0000 7fee49d3b640 1 -- start start 2024-03-29T11:22:59.386 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.408+0000 7fee49d3b640 10 monclient: init 2024-03-29T11:22:59.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.410+0000 7fee49d3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:59.387 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.410+0000 7fee49d3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:59.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.410+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fee3c0389e0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:59.389 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fee3c038dc0 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fee3c030660 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c0c54c0 con 0x7fee3c030660 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c036430 con 0x7fee3c0389e0 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c039840 con 0x7fee3c038dc0 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee49d3b640 10 monclient(hunting): authenticate will time out at 1306.666577s 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee4953a640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee3c030660 0x7fee3c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38980/0 (socket says 172.21.15.3:38980) 2024-03-29T11:22:59.390 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.412+0000 7fee4953a640 1 -- 172.21.15.3:0/2639351853 learned_addr learned my addr 172.21.15.3:0/2639351853 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.414+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262429714 0 0) 0x7fee3c0c54c0 con 0x7fee3c030660 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee34003520 con 0x7fee3c030660 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961079689 0 0) 0x7fee3c036430 con 0x7fee3c0389e0 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.395 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.415+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c0c54c0 con 0x7fee3c0389e0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073728223 0 0) 0x7fee3c0c54c0 con 0x7fee3c0389e0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 >> v1:172.21.15.3:6790/0 conn(0x7fee3c038dc0 legacy=0x7fee3c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 >> v1:172.21.15.3:6789/0 conn(0x7fee3c030660 legacy=0x7fee3c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 monclient: found mon.noname-c 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c035690 con 0x7fee3c0389e0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee30001670 con 0x7fee3c0389e0 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.396 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee437fe640 10 monclient: dump: 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee49d3b640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.416+0000 7fee49d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:59.397 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee300017d0 con 0x7fee3c0389e0 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 1 -- 172.21.15.3:0/2639351853 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee30001c20 con 0x7fee3c0389e0 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee49d3b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee49d3b640 4 set_mon_vals no callback set 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.417+0000 7fee437fe640 10 monclient: dump: 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.398 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.399 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.399 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:59.399 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:59.399 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:59.399 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 10 monclient: shutdown 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 1 -- 172.21.15.3:0/2639351853 >> v1:172.21.15.32:6789/0 conn(0x7fee3c0389e0 legacy=0x7fee3c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 1 -- 172.21.15.3:0/2639351853 shutdown_connections 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 1 -- 172.21.15.3:0/2639351853 >> 172.21.15.3:0/2639351853 conn(0x7fee3c005c00 msgr2=0x7fee3c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 1 -- 172.21.15.3:0/2639351853 shutdown_connections 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.418+0000 7fee49d3b640 1 -- 172.21.15.3:0/2639351853 wait complete. 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.419+0000 7fee49d3b640 10 monclient: build_initial_monmap 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.419+0000 7fee49d3b640 10 monclient: monmap: 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:22:59.403 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.420+0000 7fee49d3b640 1 librados: starting msgr at 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.420+0000 7fee49d3b640 1 librados: starting objecter 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.420+0000 7fee49d3b640 1 Processor -- start 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.422+0000 7fee49d3b640 1 -- start start 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.422+0000 7fee49d3b640 1 librados: setting wanted keys 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.422+0000 7fee49d3b640 1 librados: calling monclient init 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.422+0000 7fee49d3b640 10 monclient: init 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:59.404 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-c con 0x7fee3c030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-a con 0x7fee3c0389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient(hunting): picked mon.noname-b con 0x7fee3c15cea0 addr v1:172.21.15.32:6789/0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c02f440 con 0x7fee3c0389e0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee43fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fee3c0389e0 0x7fee3c0c22c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:40682/0 (socket says 172.21.15.3:40682) 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee43fff640 1 -- 172.21.15.3:0/2110731013 learned_addr learned my addr 172.21.15.3:0/2110731013 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c02ea10 con 0x7fee3c15cea0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee417fa640 10 client.?.objecter ms_handle_connect 0x7fee3c0389e0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3c02f0b0 con 0x7fee3c030660 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.423+0000 7fee49d3b640 10 monclient(hunting): authenticate will time out at 1306.677872s 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995054145 0 0) 0x7fee3c02f440 con 0x7fee3c0389e0 2024-03-29T11:22:59.405 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee28004eb0 con 0x7fee3c0389e0 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 914388099 0 0) 0x7fee28004eb0 con 0x7fee3c0389e0 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee3c02f440 con 0x7fee3c0389e0 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee4404fdd0 con 0x7fee3c0389e0 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): dump: 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:59.406 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1517885257 0 0) 0x7fee3c02f440 con 0x7fee3c0389e0 2024-03-29T11:22:59.407 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.424+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 >> v1:172.21.15.3:6790/0 conn(0x7fee3c030660 legacy=0x7fee3c15c790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 >> v1:172.21.15.32:6789/0 conn(0x7fee3c15cea0 legacy=0x7fee3c1680b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 1 monclient: found mon.a 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3c02f960 con 0x7fee3c0389e0 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 client.?.objecter ms_handle_connect 0x7fee3c15cea0 2024-03-29T11:22:59.411 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee49d3b640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee49d3b640 10 monclient: _renew_subs 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee49d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.432+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee3c02ecf0 con 0x7fee3c0389e0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee49d3b640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee49d3b640 10 monclient: _renew_subs 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee49d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee3c030180 con 0x7fee3c0389e0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee49d3b640 1 librados: init done 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee4404f430 con 0x7fee3c0389e0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fee44075f80 con 0x7fee3c0389e0 2024-03-29T11:22:59.412 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee417fa640 10 monclient: dump: 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:22:59.413 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.433+0000 7fee42ffd640 4 set_mon_vals no callback set 2024-03-29T11:22:59.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.434+0000 7fee49d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.434+0000 7fee49d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.434+0000 7fee49d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:59.414 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.434+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee24005080 con 0x7fee3c0389e0 2024-03-29T11:22:59.416 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.438+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fee44067be0 con 0x7fee3c0389e0 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fee440ceb00 con 0x7fee3c0389e0 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee417fa640 10 client.4518.objecter ms_dispatch 0x7fee44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee417fa640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee417fa640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee417fa640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.439+0000 7fee48d39640 10 monclient: get_auth_request con 0x7fee2806dc70 auth_method 0 2024-03-29T11:22:59.417 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.440+0000 7fee417fa640 10 client.4518.objecter ms_handle_connect 0x7fee2806dc70 2024-03-29T11:22:59.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.450+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee440cb4a0 con 0x7fee3c0389e0 2024-03-29T11:22:59.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.450+0000 7fee417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:59.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.450+0000 7fee417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.792+0000 7fee49d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.793+0000 7fee49d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.793+0000 7fee49d3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:22:59.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.793+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee24005330 con 0x7fee3c0389e0 2024-03-29T11:22:59.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.794+0000 7fee417fa640 1 -- 172.21.15.3:0/2110731013 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fee440cb4a0 con 0x7fee3c0389e0 2024-03-29T11:22:59.772 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:22:59.772 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:59.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.794+0000 7fee417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:59.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.794+0000 7fee417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:59.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.797+0000 7fee49d3b640 10 librados: watch_flush enter 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.797+0000 7fee49d3b640 10 librados: watch_flush exit 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.797+0000 7fee49d3b640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.797+0000 7fee49d3b640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.798+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 >> v1:172.21.15.3:6816/844689376 conn(0x7fee2806dc70 legacy=0x7fee28070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.798+0000 7fee49d3b640 10 monclient: shutdown 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.798+0000 7fee49d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:59.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.798+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 >> v1:172.21.15.3:6789/0 conn(0x7fee3c0389e0 legacy=0x7fee3c0c22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:59.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.799+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 shutdown_connections 2024-03-29T11:22:59.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.799+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 >> 172.21.15.3:0/2110731013 conn(0x7fee3c005c00 msgr2=0x7fee3c034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:59.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.799+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 shutdown_connections 2024-03-29T11:22:59.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.799+0000 7fee49d3b640 1 -- 172.21.15.3:0/2110731013 wait complete. 2024-03-29T11:22:59.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:22:59.799+0000 7fee49d3b640 1 librados: shutdown 2024-03-29T11:22:59.795 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:06.602 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:06.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.771+0000 7f308c4e6640 10 monclient: get_monmap_and_config 2024-03-29T11:23:06.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.771+0000 7f308c4e6640 10 monclient: build_initial_monmap 2024-03-29T11:23:06.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.771+0000 7f308c4e6640 10 monclient: monmap: 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:06.772253+0000 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:06.772253+0000 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:06.750 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.772+0000 7f308c4e6640 1 Processor -- start 2024-03-29T11:23:06.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.772+0000 7f308c4e6640 1 -- start start 2024-03-29T11:23:06.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.772+0000 7f308c4e6640 10 monclient: init 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.773+0000 7f308c4e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.773+0000 7f308c4e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.773+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f30840a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.773+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f30840aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f3084146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30840aed20 con 0x7f30840a98a0 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30840aa8b0 con 0x7f30840aab00 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084146a60 con 0x7f3084146f80 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:06.754 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308c4e6640 10 monclient(hunting): authenticate will time out at 1314.028075s 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308aa83640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3084146f80 0x7f3084151cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:32960/0 (socket says 172.21.15.3:32960) 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.774+0000 7f308aa83640 1 -- 172.21.15.3:0/1726047992 learned_addr learned my addr 172.21.15.3:0/1726047992 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877807378 0 0) 0x7f3084146a60 con 0x7f3084146f80 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.775+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003520 con 0x7f3084146f80 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746300358 0 0) 0x7f30840aa8b0 con 0x7f30840aab00 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:23:06.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003840 con 0x7f30840aab00 2024-03-29T11:23:06.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999981222 0 0) 0x7f30840aed20 con 0x7f30840a98a0 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3070003b80 con 0x7f30840a98a0 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2745869907 0 0) 0x7f3070003840 con 0x7f30840aab00 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 >> v1:172.21.15.3:6790/0 conn(0x7f3084146f80 legacy=0x7f3084151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 >> v1:172.21.15.3:6789/0 conn(0x7f30840a98a0 legacy=0x7f30840ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: _finish_hunting 0 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 monclient: found mon.noname-c 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084143ef0 con 0x7f30840aab00 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: _finish_auth 0 2024-03-29T11:23:06.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: _check_auth_tickets 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f307c001970 con 0x7f30840aab00 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f308c4e6640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f308c4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f3089280640 10 monclient: dump: 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.758 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.776+0000 7f308c4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.777+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f307c001ad0 con 0x7f30840aab00 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f3089280640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f3089280640 1 -- 172.21.15.3:0/1726047992 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f307c001f20 con 0x7f30840aab00 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f3089280640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f3089280640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f3089280640 10 monclient: dump: 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.759 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f308c4e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.778+0000 7f308c4e6640 4 set_mon_vals no callback set 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.779+0000 7f308c4e6640 10 monclient: shutdown 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.779+0000 7f308c4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:06.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.779+0000 7f308c4e6640 1 -- 172.21.15.3:0/1726047992 >> v1:172.21.15.32:6789/0 conn(0x7f30840aab00 legacy=0x7f30841437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:06.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.779+0000 7f308c4e6640 1 -- 172.21.15.3:0/1726047992 shutdown_connections 2024-03-29T11:23:06.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.779+0000 7f308c4e6640 1 -- 172.21.15.3:0/1726047992 >> 172.21.15.3:0/1726047992 conn(0x7f30840b3c10 msgr2=0x7f30840b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:06.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.780+0000 7f308c4e6640 1 -- 172.21.15.3:0/1726047992 shutdown_connections 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.780+0000 7f308c4e6640 1 -- 172.21.15.3:0/1726047992 wait complete. 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.781+0000 7f308c4e6640 10 monclient: build_initial_monmap 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.781+0000 7f308c4e6640 10 monclient: monmap: 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 librados: starting msgr at 2024-03-29T11:23:06.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 librados: starting objecter 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 Processor -- start 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 -- start start 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 librados: setting wanted keys 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 1 librados: calling monclient init 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.782+0000 7f308c4e6640 10 monclient: init 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308c4e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308c4e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f30840a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f30840aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308a282640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f30840a98a0 0x7f30841489c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39928/0 (socket says 172.21.15.3:39928) 2024-03-29T11:23:06.767 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308a282640 1 -- 172.21.15.3:0/1782439094 learned_addr learned my addr 172.21.15.3:0/1782439094 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f308c4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f3084146f80 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f307affd640 10 client.?.objecter ms_handle_connect 0x7f30840aab00 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f307affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f307affd640 10 client.?.objecter ms_handle_connect 0x7f30840a98a0 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.783+0000 7f307affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3084139720 con 0x7f30840a98a0 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308413b450 con 0x7f3084146f80 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f308413b730 con 0x7f30840aab00 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f308c4e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f308c4e6640 10 monclient(hunting): authenticate will time out at 1314.038189s 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 client.?.objecter ms_handle_connect 0x7f3084146f80 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:06.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675051115 0 0) 0x7f308413b450 con 0x7f3084146f80 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.784+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068004eb0 con 0x7f3084146f80 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693107209 0 0) 0x7f308413b730 con 0x7f30840aab00 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308413b450 con 0x7f30840aab00 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493720029 0 0) 0x7f3084139720 con 0x7f30840a98a0 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:06.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308413b730 con 0x7f30840a98a0 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209627527 0 0) 0x7f3068004eb0 con 0x7f3084146f80 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3084139720 con 0x7f3084146f80 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f30800017b0 con 0x7f3084146f80 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): dump: 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.770 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.785+0000 7f307affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2536785492 0 0) 0x7f308413b450 con 0x7f30840aab00 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3068004eb0 con 0x7f30840aab00 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f307c001de0 con 0x7f30840aab00 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient(hunting): dump: 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.771 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 336611880 0 0) 0x7f3084139720 con 0x7f3084146f80 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 >> v1:172.21.15.3:6790/0 conn(0x7f30840aab00 legacy=0x7f308413bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 >> v1:172.21.15.3:6789/0 conn(0x7f30840a98a0 legacy=0x7f30841489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 monclient: found mon.b 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3084139440 con 0x7f3084146f80 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:06.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f307affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f308c4e6640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f308c4e6640 10 monclient: _renew_subs 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f308c4e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.786+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308413c970 con 0x7f3084146f80 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f308c4e6640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f308c4e6640 10 monclient: _renew_subs 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f308c4e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30841492b0 con 0x7f3084146f80 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f308c4e6640 1 librados: init done 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3080001ca0 con 0x7f3084146f80 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3080004980 con 0x7f3084146f80 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f307affd640 10 monclient: dump: 2024-03-29T11:23:06.773 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:06.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.787+0000 7f3088a7f640 4 set_mon_vals no callback set 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.789+0000 7f308c4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.789+0000 7f308c4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.789+0000 7f308c4e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.789+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f308406a400 con 0x7f3084146f80 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.789+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f308001a950 con 0x7f3084146f80 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.790+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f3080084830 con 0x7f3084146f80 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.790+0000 7f308a282640 10 monclient: get_auth_request con 0x7f306806e050 auth_method 0 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.790+0000 7f307affd640 10 client.4534.objecter ms_dispatch 0x7f3084000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:06.776 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.790+0000 7f307affd640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.790+0000 7f307affd640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.791+0000 7f307affd640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.791+0000 7f307affd640 10 client.4534.objecter ms_handle_connect 0x7f306806e050 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.797+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3080050110 con 0x7f3084146f80 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.797+0000 7f307affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:06.777 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:06.797+0000 7f307affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.138+0000 7f308c4e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.138+0000 7f308c4e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.138+0000 7f308c4e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.138+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f308406ab60 con 0x7f3084146f80 2024-03-29T11:23:07.116 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.139+0000 7f307affd640 1 -- 172.21.15.3:0/1782439094 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f3080052040 con 0x7f3084146f80 2024-03-29T11:23:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.139+0000 7f307affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:07.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.139+0000 7f307affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:07.118 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:07.118 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:07.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.144+0000 7f308c4e6640 10 librados: watch_flush enter 2024-03-29T11:23:07.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.144+0000 7f308c4e6640 10 librados: watch_flush exit 2024-03-29T11:23:07.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.144+0000 7f308c4e6640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-29T11:23:07.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.144+0000 7f308c4e6640 10 client.4534.objecter successfully canceled tick 2024-03-29T11:23:07.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.145+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 >> v1:172.21.15.3:6816/844689376 conn(0x7f306806e050 legacy=0x7f3068070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.145+0000 7f308c4e6640 10 monclient: shutdown 2024-03-29T11:23:07.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.145+0000 7f308c4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:07.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.145+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 >> v1:172.21.15.32:6789/0 conn(0x7f3084146f80 legacy=0x7f308413c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:07.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.146+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 shutdown_connections 2024-03-29T11:23:07.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.146+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 >> 172.21.15.3:0/1782439094 conn(0x7f30840b3c10 msgr2=0x7f30840ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:07.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.146+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 shutdown_connections 2024-03-29T11:23:07.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.146+0000 7f308c4e6640 1 -- 172.21.15.3:0/1782439094 wait complete. 2024-03-29T11:23:07.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:07.146+0000 7f308c4e6640 1 librados: shutdown 2024-03-29T11:23:07.138 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:13.944 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:14.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.132+0000 7f6f006eb640 10 monclient: get_monmap_and_config 2024-03-29T11:23:14.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.132+0000 7f6f006eb640 10 monclient: build_initial_monmap 2024-03-29T11:23:14.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.132+0000 7f6f006eb640 10 monclient: monmap: 2024-03-29T11:23:14.111 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:14.133494+0000 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:14.133494+0000 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.112 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:14.113 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:14.113 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:14.113 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.133+0000 7f6f006eb640 1 Processor -- start 2024-03-29T11:23:14.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.133+0000 7f6f006eb640 1 -- start start 2024-03-29T11:23:14.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.133+0000 7f6f006eb640 10 monclient: init 2024-03-29T11:23:14.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:14.121 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef80b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef80b2cb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef80b3560 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef80aab60 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef80aae40 con 0x7f6ef80b3560 2024-03-29T11:23:14.122 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef80ab120 con 0x7f6ef80b28d0 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6f006eb640 10 monclient(hunting): authenticate will time out at 1321.389156s 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6efdc86640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f6ef80b2cb0 0x7f6ef8142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54840/0 (socket says 172.21.15.3:54840) 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.135+0000 7f6efdc86640 1 -- 172.21.15.3:0/903857537 learned_addr learned my addr 172.21.15.3:0/903857537 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421772352 0 0) 0x7f6ef80aab60 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _negotiate 2024-03-29T11:23:14.123 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee4003520 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322963120 0 0) 0x7f6ef80aae40 con 0x7f6ef80b3560 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _negotiate 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:14.124 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef80aab60 con 0x7f6ef80b3560 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1698243741 0 0) 0x7f6ee4003520 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 >> v1:172.21.15.3:6790/0 conn(0x7f6ef80b28d0 legacy=0x7f6ef813bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 >> v1:172.21.15.32:6789/0 conn(0x7f6ef80b3560 legacy=0x7f6ef81428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 10 monclient: _finish_hunting 0 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.136+0000 7f6efd485640 1 monclient: found mon.noname-a 2024-03-29T11:23:14.125 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef813c380 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: _finish_auth 0 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: _check_auth_tickets 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6f006eb640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:23:14.126 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6f006eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6eec001690 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: dump: 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:14.127 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.128 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6f006eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6eec001870 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 1 -- 172.21.15.3:0/903857537 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6eec001c80 con 0x7f6ef80b2cb0 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6efd485640 10 monclient: dump: 2024-03-29T11:23:14.129 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:14.130 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6f006eb640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.137+0000 7f6f006eb640 4 set_mon_vals no callback set 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.138+0000 7f6f006eb640 10 monclient: shutdown 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.138+0000 7f6f006eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:14.131 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.138+0000 7f6f006eb640 1 -- 172.21.15.3:0/903857537 >> v1:172.21.15.3:6789/0 conn(0x7f6ef80b2cb0 legacy=0x7f6ef8142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.139+0000 7f6f006eb640 1 -- 172.21.15.3:0/903857537 shutdown_connections 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.139+0000 7f6f006eb640 1 -- 172.21.15.3:0/903857537 >> 172.21.15.3:0/903857537 conn(0x7f6ef8066a00 msgr2=0x7f6ef8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.139+0000 7f6f006eb640 1 -- 172.21.15.3:0/903857537 shutdown_connections 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.139+0000 7f6f006eb640 1 -- 172.21.15.3:0/903857537 wait complete. 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 10 monclient: build_initial_monmap 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 10 monclient: monmap: 2024-03-29T11:23:14.132 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:14.133 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 librados: starting msgr at 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 librados: starting objecter 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 Processor -- start 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 -- start start 2024-03-29T11:23:14.134 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 librados: setting wanted keys 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 1 librados: calling monclient init 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.140+0000 7f6f006eb640 10 monclient: init 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef80b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef813f370 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef8070c80 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:14.135 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef806e240 con 0x7f6ef80b28d0 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef806ef90 con 0x7f6ef8070c80 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ef806f270 con 0x7f6ef813f370 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.141+0000 7f6f006eb640 10 monclient(hunting): authenticate will time out at 1321.395805s 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6efec88640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f6ef8070c80 0x7f6ef81e6db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:51202/0 (socket says 172.21.15.3:51202) 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6efec88640 1 -- 172.21.15.3:0/706856452 learned_addr learned my addr 172.21.15.3:0/706856452 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:14.136 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter ms_handle_connect 0x7f6ef8070c80 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter ms_handle_connect 0x7f6ef80b28d0 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter ms_handle_connect 0x7f6ef813f370 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723109881 0 0) 0x7f6ef806ef90 con 0x7f6ef8070c80 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:14.137 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:14.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:14.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:23:14.138 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.142+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc004eb0 con 0x7f6ef8070c80 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 153525644 0 0) 0x7f6edc004eb0 con 0x7f6ef8070c80 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ef806ef90 con 0x7f6ef8070c80 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6ef0001c30 con 0x7f6ef8070c80 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:14.143 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 10 monclient(hunting): dump: 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:14.144 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.143+0000 7f6eeaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676996826 0 0) 0x7f6ef806e240 con 0x7f6ef80b28d0 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:14.145 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6edc004eb0 con 0x7f6ef80b28d0 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 179006553 0 0) 0x7f6ef806ef90 con 0x7f6ef8070c80 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 >> v1:172.21.15.3:6790/0 conn(0x7f6ef813f370 legacy=0x7f6ef8132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 >> v1:172.21.15.3:6789/0 conn(0x7f6ef80b28d0 legacy=0x7f6ef8145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.146 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 monclient: found mon.b 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef806de80 con 0x7f6ef8070c80 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:14.147 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 10 monclient: _renew_subs 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ef8144860 con 0x7f6ef8070c80 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:14.148 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 10 monclient: _renew_subs 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ef80712b0 con 0x7f6ef8070c80 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6f006eb640 1 librados: init done 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ef0003990 con 0x7f6ef8070c80 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6eeaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.144+0000 7f6efcc84640 4 set_mon_vals no callback set 2024-03-29T11:23:14.149 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.145+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6ef0003d60 con 0x7f6ef8070c80 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.145+0000 7f6eeaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.145+0000 7f6eeaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.145+0000 7f6eeaffd640 10 monclient: dump: 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.150 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:14.151 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.146+0000 7f6f006eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.146+0000 7f6f006eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.146+0000 7f6f006eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.146+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0005080 con 0x7f6ef8070c80 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.147+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f6ef0003ec0 con 0x7f6ef8070c80 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f6ef0084d50 con 0x7f6ef8070c80 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6efe487640 10 monclient: get_auth_request con 0x7f6edc06dd70 auth_method 0 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 10 client.4540.objecter ms_dispatch 0x7f6ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:14.152 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 3 client.4540.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 3 client.4540.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 20 client.4540.objecter dump_active .. 0 homeless 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.148+0000 7f6eeaffd640 10 client.4540.objecter ms_handle_connect 0x7f6edc06dd70 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.154+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ef0050630 con 0x7f6ef8070c80 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.154+0000 7f6eeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:14.153 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.154+0000 7f6eeaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:14.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.575+0000 7f6f006eb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.575+0000 7f6f006eb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.575+0000 7f6f006eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.575+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ee0005330 con 0x7f6ef8070c80 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.576+0000 7f6eeaffd640 1 -- 172.21.15.3:0/706856452 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f6ef0052560 con 0x7f6ef8070c80 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.576+0000 7f6eeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:14.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.576+0000 7f6eeaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:14.555 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:14.555 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.579+0000 7f6f006eb640 10 librados: watch_flush enter 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 10 librados: watch_flush exit 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 20 client.4540.objecter shutdown clearing up homeless session... 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 10 client.4540.objecter successfully canceled tick 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 >> v1:172.21.15.3:6816/844689376 conn(0x7f6edc06dd70 legacy=0x7f6edc070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 10 monclient: shutdown 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:14.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 >> v1:172.21.15.32:6789/0 conn(0x7f6ef8070c80 legacy=0x7f6ef81e6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:14.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 shutdown_connections 2024-03-29T11:23:14.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 >> 172.21.15.3:0/706856452 conn(0x7f6ef8066a00 msgr2=0x7f6ef81357c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:14.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 shutdown_connections 2024-03-29T11:23:14.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 -- 172.21.15.3:0/706856452 wait complete. 2024-03-29T11:23:14.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:14.580+0000 7f6f006eb640 1 librados: shutdown 2024-03-29T11:23:14.573 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:21.380 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.550+0000 7fa92923a640 10 monclient: get_monmap_and_config 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.550+0000 7fa92923a640 10 monclient: build_initial_monmap 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.550+0000 7fa92923a640 10 monclient: monmap: 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:21.551465+0000 2024-03-29T11:23:21.529 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:21.551465+0000 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:21.530 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.553+0000 7fa92923a640 1 Processor -- start 2024-03-29T11:23:21.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.553+0000 7fa92923a640 1 -- start start 2024-03-29T11:23:21.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.553+0000 7fa92923a640 10 monclient: init 2024-03-29T11:23:21.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:21.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:21.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-a con 0x7fa91c0389e0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:21.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-c con 0x7fa91c038dc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-b con 0x7fa91c030660 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0c54c0 con 0x7fa91c0389e0 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c036430 con 0x7fa91c038dc0 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c039840 con 0x7fa91c030660 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.555+0000 7fa92923a640 10 monclient(hunting): authenticate will time out at 1328.809943s 2024-03-29T11:23:21.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.556+0000 7fa923fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa91c0389e0 0x7fa91c033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:53850/0 (socket says 172.21.15.3:53850) 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.556+0000 7fa923fff640 1 -- 172.21.15.3:0/3233839779 learned_addr learned my addr 172.21.15.3:0/3233839779 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.556+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308754498 0 0) 0x7fa91c0c54c0 con 0x7fa91c0389e0 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914003520 con 0x7fa91c0389e0 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 872635935 0 0) 0x7fa914003520 con 0x7fa91c0389e0 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 >> v1:172.21.15.3:6790/0 conn(0x7fa91c030660 legacy=0x7fa91c0d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 >> v1:172.21.15.32:6789/0 conn(0x7fa91c038dc0 legacy=0x7fa91c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 monclient: found mon.noname-a 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c035690 con 0x7fa91c0389e0 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa918001690 con 0x7fa91c0389e0 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: dump: 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.540 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa92923a640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa92923a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa918001860 con 0x7fa91c0389e0 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 1 -- 172.21.15.3:0/3233839779 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa918001cb0 con 0x7fa91c0389e0 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa922ffd640 10 monclient: dump: 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.541 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa92923a640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.557+0000 7fa92923a640 4 set_mon_vals no callback set 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 10 monclient: shutdown 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 1 -- 172.21.15.3:0/3233839779 >> v1:172.21.15.3:6789/0 conn(0x7fa91c0389e0 legacy=0x7fa91c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 1 -- 172.21.15.3:0/3233839779 shutdown_connections 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 1 -- 172.21.15.3:0/3233839779 >> 172.21.15.3:0/3233839779 conn(0x7fa91c005c00 msgr2=0x7fa91c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:21.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.558+0000 7fa92923a640 1 -- 172.21.15.3:0/3233839779 shutdown_connections 2024-03-29T11:23:21.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.559+0000 7fa92923a640 1 -- 172.21.15.3:0/3233839779 wait complete. 2024-03-29T11:23:21.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.559+0000 7fa92923a640 10 monclient: build_initial_monmap 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.559+0000 7fa92923a640 10 monclient: monmap: 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 librados: starting msgr at 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 librados: starting objecter 2024-03-29T11:23:21.545 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 Processor -- start 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 -- start start 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 librados: setting wanted keys 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 1 librados: calling monclient init 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.560+0000 7fa92923a640 10 monclient: init 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.561+0000 7fa92923a640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.562+0000 7fa92923a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.562+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-a con 0x7fa91c030660 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.562+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-c con 0x7fa91c0389e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.562+0000 7fa92923a640 10 monclient(hunting): picked mon.noname-b con 0x7fa91c0c26d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa92923a640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0c8190 con 0x7fa91c030660 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa92923a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0c7c70 con 0x7fa91c0c26d0 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa92923a640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91c0c2c40 con 0x7fa91c0389e0 2024-03-29T11:23:21.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa92923a640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:21.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa92923a640 10 monclient(hunting): authenticate will time out at 1328.818235s 2024-03-29T11:23:21.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa923fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa91c030660 0x7fa91c15c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:54222/0 (socket says 172.21.15.3:54222) 2024-03-29T11:23:21.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa923fff640 1 -- 172.21.15.3:0/104635909 learned_addr learned my addr 172.21.15.3:0/104635909 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:21.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa920ff9640 10 client.?.objecter ms_handle_connect 0x7fa91c030660 2024-03-29T11:23:21.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.564+0000 7fa920ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 client.?.objecter ms_handle_connect 0x7fa91c0389e0 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614931120 0 0) 0x7fa91c0c8190 con 0x7fa91c030660 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa908004eb0 con 0x7fa91c030660 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170814766 0 0) 0x7fa91c0c2c40 con 0x7fa91c0389e0 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:23:21.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.567+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa91c0c8190 con 0x7fa91c0389e0 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2271835697 0 0) 0x7fa908004eb0 con 0x7fa91c030660 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa91c0c2c40 con 0x7fa91c030660 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa918001b70 con 0x7fa91c030660 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:21.554 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 client.?.objecter ms_handle_connect 0x7fa91c0c26d0 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968750984 0 0) 0x7fa91c0c2c40 con 0x7fa91c030660 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 >> v1:172.21.15.3:6790/0 conn(0x7fa91c0389e0 legacy=0x7fa91c0c1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 >> v1:172.21.15.32:6789/0 conn(0x7fa91c0c26d0 legacy=0x7fa91c1682b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 monclient: found mon.a 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91c0c7820 con 0x7fa91c030660 2024-03-29T11:23:21.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 5 monclient: authenticate success, global_id 4536 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 10 monclient: _renew_subs 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91c15d0a0 con 0x7fa91c030660 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 10 monclient: _renew_subs 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91c0c7280 con 0x7fa91c030660 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa92923a640 1 librados: init done 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa918004bf0 con 0x7fa91c030660 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa918001b70 con 0x7fa91c030660 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:21.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa920ff9640 10 monclient: dump: 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.568+0000 7fa9227fc640 4 set_mon_vals no callback set 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.570+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fa91801abd0 con 0x7fa91c030660 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa92923a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:21.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa92923a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa92923a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa904005080 con 0x7fa91c030660 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fa9180848d0 con 0x7fa91c030660 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa920ff9640 10 client.4536.objecter ms_dispatch 0x7fa924000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa920ff9640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa920ff9640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa920ff9640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.571+0000 7fa9237fe640 10 monclient: get_auth_request con 0x7fa90806de90 auth_method 0 2024-03-29T11:23:21.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.572+0000 7fa920ff9640 10 client.4536.objecter ms_handle_connect 0x7fa90806de90 2024-03-29T11:23:21.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.581+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa918050210 con 0x7fa91c030660 2024-03-29T11:23:21.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.581+0000 7fa920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:21.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.581+0000 7fa920ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:21.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.909+0000 7fa92923a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:21.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.910+0000 7fa92923a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:21.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.910+0000 7fa92923a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:21.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.910+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa904005330 con 0x7fa91c030660 2024-03-29T11:23:21.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.911+0000 7fa920ff9640 1 -- 172.21.15.3:0/104635909 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fa918050210 con 0x7fa91c030660 2024-03-29T11:23:21.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.911+0000 7fa920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:21.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.911+0000 7fa920ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:21.890 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:21.890 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:21.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 10 librados: watch_flush enter 2024-03-29T11:23:21.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 10 librados: watch_flush exit 2024-03-29T11:23:21.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-29T11:23:21.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 10 client.4536.objecter successfully canceled tick 2024-03-29T11:23:21.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 >> v1:172.21.15.3:6816/844689376 conn(0x7fa90806de90 legacy=0x7fa908070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 10 monclient: shutdown 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 >> v1:172.21.15.3:6789/0 conn(0x7fa91c030660 legacy=0x7fa91c15c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 shutdown_connections 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 >> 172.21.15.3:0/104635909 conn(0x7fa91c005c00 msgr2=0x7fa91c0c3760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 shutdown_connections 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 -- 172.21.15.3:0/104635909 wait complete. 2024-03-29T11:23:21.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:21.915+0000 7fa92923a640 1 librados: shutdown 2024-03-29T11:23:21.908 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:28.715 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:28.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.888+0000 7f3d8f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:28.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.888+0000 7f3d8f59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:28.867 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.888+0000 7f3d8f59e640 10 monclient: monmap: 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:28.889538+0000 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:28.889538+0000 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:28.868 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.889+0000 7f3d8f59e640 1 Processor -- start 2024-03-29T11:23:28.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.889+0000 7f3d8f59e640 1 -- start start 2024-03-29T11:23:28.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.889+0000 7f3d8f59e640 10 monclient: init 2024-03-29T11:23:28.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:28.869 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3d900af800 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3d900b1cd0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3d90145cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900b0a00 con 0x7f3d900af800 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900b17b0 con 0x7f3d900b1cd0 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d900aed60 con 0x7f3d90145cb0 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.890+0000 7f3d8f59e640 10 monclient(hunting): authenticate will time out at 1336.144772s 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.891+0000 7f3d87fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d900b1cd0 0x7f3d9013f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:51332/0 (socket says 172.21.15.3:51332) 2024-03-29T11:23:28.870 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.891+0000 7f3d87fff640 1 -- 172.21.15.3:0/2839786173 learned_addr learned my addr 172.21.15.3:0/2839786173 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271573547 0 0) 0x7f3d900b17b0 con 0x7f3d900b1cd0 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:23:28.871 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.892+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d70003520 con 0x7f3d900b1cd0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.893+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1352728601 0 0) 0x7f3d70003520 con 0x7f3d900b1cd0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 >> v1:172.21.15.3:6790/0 conn(0x7f3d90145cb0 legacy=0x7f3d9014d8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 >> v1:172.21.15.3:6789/0 conn(0x7f3d900af800 legacy=0x7f3d901455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d900aeef0 con 0x7f3d900b1cd0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:28.873 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3d78001690 con 0x7f3d900b1cd0 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8dd9b640 10 monclient: dump: 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:28.874 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:28.875 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8f59e640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:23:28.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.894+0000 7f3d8f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d780017f0 con 0x7f3d900b1cd0 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 1 -- 172.21.15.3:0/2839786173 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3d78001c40 con 0x7f3d900b1cd0 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8dd9b640 10 monclient: dump: 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.876 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.895+0000 7f3d8f59e640 4 set_mon_vals no callback set 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.896+0000 7f3d8f59e640 10 monclient: shutdown 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.896+0000 7f3d8f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.896+0000 7f3d8f59e640 1 -- 172.21.15.3:0/2839786173 >> v1:172.21.15.32:6789/0 conn(0x7f3d900b1cd0 legacy=0x7f3d9013f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.897+0000 7f3d8f59e640 1 -- 172.21.15.3:0/2839786173 shutdown_connections 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.897+0000 7f3d8f59e640 1 -- 172.21.15.3:0/2839786173 >> 172.21.15.3:0/2839786173 conn(0x7f3d90066a00 msgr2=0x7f3d90066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.897+0000 7f3d8f59e640 1 -- 172.21.15.3:0/2839786173 shutdown_connections 2024-03-29T11:23:28.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.897+0000 7f3d8f59e640 1 -- 172.21.15.3:0/2839786173 wait complete. 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.898+0000 7f3d8f59e640 10 monclient: build_initial_monmap 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.898+0000 7f3d8f59e640 10 monclient: monmap: 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:28.878 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 librados: starting msgr at 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 librados: starting objecter 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 Processor -- start 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 -- start start 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 librados: setting wanted keys 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 1 librados: calling monclient init 2024-03-29T11:23:28.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.899+0000 7f3d8f59e640 10 monclient: init 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f3d900af800 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f3d900b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f3d90071e00 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d90070ae0 con 0x7f3d900af800 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9006fcd0 con 0x7f3d90071e00 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9006ffb0 con 0x7f3d900b1cd0 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8f59e640 10 monclient(hunting): authenticate will time out at 1336.154192s 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8ed9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d90071e00 0x7f3d901e61c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:51846/0 (socket says 172.21.15.3:51846) 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d8ed9d640 1 -- 172.21.15.3:0/1987341752 learned_addr learned my addr 172.21.15.3:0/1987341752 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d86ffd640 10 client.?.objecter ms_handle_connect 0x7f3d90071e00 2024-03-29T11:23:28.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.900+0000 7f3d86ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311774272 0 0) 0x7f3d9006fcd0 con 0x7f3d90071e00 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.901+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d74004eb0 con 0x7f3d90071e00 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1066368553 0 0) 0x7f3d74004eb0 con 0x7f3d90071e00 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d9006fcd0 con 0x7f3d90071e00 2024-03-29T11:23:28.882 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3d88001ad0 con 0x7f3d90071e00 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.902+0000 7f3d86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:28.883 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.903+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1217112072 0 0) 0x7f3d9006fcd0 con 0x7f3d90071e00 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.903+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 >> v1:172.21.15.3:6790/0 conn(0x7f3d900b1cd0 legacy=0x7f3d90140390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 >> v1:172.21.15.3:6789/0 conn(0x7f3d900af800 legacy=0x7f3d900703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 1 monclient: found mon.b 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d90139cc0 con 0x7f3d90071e00 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:28.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 10 monclient: _renew_subs 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d90070d40 con 0x7f3d90071e00 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 client.4546.objecter ms_handle_connect 0x7f3d900af800 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d86ffd640 10 client.4546.objecter resend_mon_ops 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 10 monclient: _renew_subs 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d90140c80 con 0x7f3d90071e00 2024-03-29T11:23:28.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.905+0000 7f3d8f59e640 1 librados: init done 2024-03-29T11:23:28.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.906+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d88001c30 con 0x7f3d90071e00 2024-03-29T11:23:28.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.906+0000 7f3d86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:28.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.906+0000 7f3d8d59a640 4 set_mon_vals no callback set 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3d88004d50 con 0x7f3d90071e00 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d86ffd640 10 monclient: dump: 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:28.887 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:28.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d8f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:28.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d8f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:28.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d8f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:28.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.908+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68005080 con 0x7f3d90071e00 2024-03-29T11:23:28.888 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.909+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f3d8801ad20 con 0x7f3d90071e00 2024-03-29T11:23:28.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.910+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f3d88084d30 con 0x7f3d90071e00 2024-03-29T11:23:28.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.910+0000 7f3d86ffd640 10 client.4546.objecter ms_dispatch 0x7f3d90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:28.889 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.910+0000 7f3d86ffd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:28.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.910+0000 7f3d86ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:28.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.910+0000 7f3d86ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:23:28.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.911+0000 7f3d8e59c640 10 monclient: get_auth_request con 0x7f3d7406db10 auth_method 0 2024-03-29T11:23:28.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.911+0000 7f3d86ffd640 10 client.4546.objecter ms_handle_connect 0x7f3d7406db10 2024-03-29T11:23:28.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.918+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d88050610 con 0x7f3d90071e00 2024-03-29T11:23:28.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.918+0000 7f3d86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:28.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:28.918+0000 7f3d86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:29.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.253+0000 7f3d8f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:29.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.254+0000 7f3d8f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:29.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.254+0000 7f3d8f59e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:29.233 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.254+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d68005330 con 0x7f3d90071e00 2024-03-29T11:23:29.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.258+0000 7f3d86ffd640 1 -- 172.21.15.3:0/1987341752 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f3d88052540 con 0x7f3d90071e00 2024-03-29T11:23:29.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.258+0000 7f3d86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:29.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.258+0000 7f3d86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:29.237 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:29.237 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:29.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 10 librados: watch_flush enter 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 10 librados: watch_flush exit 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 >> v1:172.21.15.3:6816/844689376 conn(0x7f3d7406db10 legacy=0x7f3d7406ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 10 monclient: shutdown 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 >> v1:172.21.15.32:6789/0 conn(0x7f3d90071e00 legacy=0x7f3d901e61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 shutdown_connections 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 >> 172.21.15.3:0/1987341752 conn(0x7f3d90066a00 msgr2=0x7f3d900b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 shutdown_connections 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 -- 172.21.15.3:0/1987341752 wait complete. 2024-03-29T11:23:29.242 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:29.262+0000 7f3d8f59e640 1 librados: shutdown 2024-03-29T11:23:29.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:36.067 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.224+0000 7f8fa73c8640 10 monclient: get_monmap_and_config 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.224+0000 7f8fa73c8640 10 monclient: build_initial_monmap 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.224+0000 7f8fa73c8640 10 monclient: monmap: 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:36.204 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:36.225788+0000 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:36.225788+0000 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:36.205 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.225+0000 7f8fa73c8640 1 Processor -- start 2024-03-29T11:23:36.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.226+0000 7f8fa73c8640 1 -- start start 2024-03-29T11:23:36.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.226+0000 7f8fa73c8640 10 monclient: init 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa00b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa00b2cb0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa00b3560 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00aab60 con 0x7f8fa00b28d0 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00aae40 con 0x7f8fa00b3560 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00ab120 con 0x7f8fa00b2cb0 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa5965640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8fa00b3560 0x7f8fa01428a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:35516/0 (socket says 172.21.15.3:35516) 2024-03-29T11:23:36.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa5965640 1 -- 172.21.15.3:0/2189168842 learned_addr learned my addr 172.21.15.3:0/2189168842 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa73c8640 10 monclient(hunting): authenticate will time out at 1343.482927s 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.228+0000 7f8fa4963640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8fa00b2cb0 0x7f8fa0142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40116/0 (socket says 172.21.15.3:40116) 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086781280 0 0) 0x7f8fa00aae40 con 0x7f8fa00b3560 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:36.209 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f8c003520 con 0x7f8fa00b3560 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472315540 0 0) 0x7f8fa00ab120 con 0x7f8fa00b2cb0 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.229+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa00aae40 con 0x7f8fa00b2cb0 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.230+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 735241554 0 0) 0x7f8f8c003520 con 0x7f8fa00b3560 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.230+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 >> v1:172.21.15.3:6790/0 conn(0x7f8fa00b2cb0 legacy=0x7f8fa0142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:36.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.230+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 >> v1:172.21.15.3:6789/0 conn(0x7f8fa00b28d0 legacy=0x7f8fa013bc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 1 monclient: found mon.noname-c 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa013c380 con 0x7f8fa00b3560 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: _finish_auth 0 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:36.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8f98001a10 con 0x7f8fa00b3560 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8f97fff640 10 monclient: dump: 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:36.213 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8fa73c8640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:23:36.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.231+0000 7f8fa73c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:36.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f98001bf0 con 0x7f8fa00b3560 2024-03-29T11:23:36.214 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 1 -- 172.21.15.3:0/2189168842 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8f98002000 con 0x7f8fa00b3560 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8fa73c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8fa73c8640 4 set_mon_vals no callback set 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8f97fff640 10 monclient: dump: 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.215 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8fa73c8640 10 monclient: shutdown 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8fa73c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.234+0000 7f8fa73c8640 1 -- 172.21.15.3:0/2189168842 >> v1:172.21.15.32:6789/0 conn(0x7f8fa00b3560 legacy=0x7f8fa01428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.235+0000 7f8fa73c8640 1 -- 172.21.15.3:0/2189168842 shutdown_connections 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.235+0000 7f8fa73c8640 1 -- 172.21.15.3:0/2189168842 >> 172.21.15.3:0/2189168842 conn(0x7f8fa0066a00 msgr2=0x7f8fa0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.235+0000 7f8fa73c8640 1 -- 172.21.15.3:0/2189168842 shutdown_connections 2024-03-29T11:23:36.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.235+0000 7f8fa73c8640 1 -- 172.21.15.3:0/2189168842 wait complete. 2024-03-29T11:23:36.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.236+0000 7f8fa73c8640 10 monclient: build_initial_monmap 2024-03-29T11:23:36.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.236+0000 7f8fa73c8640 10 monclient: monmap: 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.237+0000 7f8fa73c8640 1 librados: starting msgr at 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.237+0000 7f8fa73c8640 1 librados: starting objecter 2024-03-29T11:23:36.218 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.238+0000 7f8fa73c8640 1 Processor -- start 2024-03-29T11:23:36.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.238+0000 7f8fa73c8640 1 -- start start 2024-03-29T11:23:36.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.238+0000 7f8fa73c8640 1 librados: setting wanted keys 2024-03-29T11:23:36.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.238+0000 7f8fa73c8640 1 librados: calling monclient init 2024-03-29T11:23:36.219 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.238+0000 7f8fa73c8640 10 monclient: init 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-a con 0x7f8fa00b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-b con 0x7f8fa012dda0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient(hunting): picked mon.noname-c con 0x7f8fa013ed50 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:36.221 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa013f820 con 0x7f8fa00b28d0 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa013f390 con 0x7f8fa012dda0 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa012d9a0 con 0x7f8fa013ed50 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa73c8640 10 monclient(hunting): authenticate will time out at 1343.493732s 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa5965640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f8fa013ed50 0x7f8fa012d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:41436/0 (socket says 172.21.15.3:41436) 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8fa5965640 1 -- 172.21.15.3:0/4100621174 learned_addr learned my addr 172.21.15.3:0/4100621174 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.239+0000 7f8f95ffb640 10 client.?.objecter ms_handle_connect 0x7f8fa013ed50 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767536945 0 0) 0x7f8fa012d9a0 con 0x7f8fa013ed50 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:36.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f84004eb0 con 0x7f8fa013ed50 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 344703653 0 0) 0x7f8f84004eb0 con 0x7f8fa013ed50 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fa012d9a0 con 0x7f8fa013ed50 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8f98001f80 con 0x7f8fa013ed50 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): dump: 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:36.223 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:36.224 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:36.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3914879122 0 0) 0x7f8fa012d9a0 con 0x7f8fa013ed50 2024-03-29T11:23:36.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 >> v1:172.21.15.32:6789/0 conn(0x7f8fa012dda0 legacy=0x7f8fa012cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:36.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.240+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 >> v1:172.21.15.3:6789/0 conn(0x7f8fa00b28d0 legacy=0x7f8fa01da720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:36.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 1 monclient: found mon.c 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa012ed20 con 0x7f8fa013ed50 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8f95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8fa73c8640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8fa73c8640 10 monclient: _renew_subs 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8fa73c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:36.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.243+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa0145f70 con 0x7f8fa013ed50 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f98004e10 con 0x7f8fa013ed50 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f8f98005220 con 0x7f8fa013ed50 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f95ffb640 10 monclient: dump: 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:36.227 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8f977fe640 4 set_mon_vals no callback set 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8fa73c8640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8fa73c8640 10 monclient: _renew_subs 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8fa73c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa01dae30 con 0x7f8fa013ed50 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.244+0000 7f8fa73c8640 1 librados: init done 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.245+0000 7f8fa73c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.245+0000 7f8fa73c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:36.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.245+0000 7f8fa73c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:36.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.245+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa006a400 con 0x7f8fa013ed50 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.246+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f8f98005380 con 0x7f8fa013ed50 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f8f98084fc0 con 0x7f8fa013ed50 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 10 client.4565.objecter ms_dispatch 0x7f8fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8fa5164640 10 monclient: get_auth_request con 0x7f8f8406da90 auth_method 0 2024-03-29T11:23:36.230 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.248+0000 7f8f95ffb640 10 client.4565.objecter ms_handle_connect 0x7f8f8406da90 2024-03-29T11:23:36.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.255+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f980508a0 con 0x7f8fa013ed50 2024-03-29T11:23:36.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.255+0000 7f8f95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:36.234 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.255+0000 7f8f95ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:36.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.566+0000 7f8fa73c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:36.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.566+0000 7f8fa73c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:36.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.566+0000 7f8fa73c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:36.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.566+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fa013f540 con 0x7f8fa013ed50 2024-03-29T11:23:36.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.568+0000 7f8f95ffb640 1 -- 172.21.15.3:0/4100621174 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f8fa013f540 con 0x7f8fa013ed50 2024-03-29T11:23:36.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.568+0000 7f8f95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:36.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.568+0000 7f8f95ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:36.548 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:36.548 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:36.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.570+0000 7f8fa73c8640 10 librados: watch_flush enter 2024-03-29T11:23:36.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.570+0000 7f8fa73c8640 10 librados: watch_flush exit 2024-03-29T11:23:36.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.570+0000 7f8fa73c8640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:23:36.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.570+0000 7f8fa73c8640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:23:36.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.571+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 >> v1:172.21.15.3:6816/844689376 conn(0x7f8f8406da90 legacy=0x7f8f8406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:36.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.571+0000 7f8fa73c8640 10 monclient: shutdown 2024-03-29T11:23:36.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.571+0000 7f8fa73c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:36.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.571+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 >> v1:172.21.15.3:6790/0 conn(0x7f8fa013ed50 legacy=0x7f8fa012d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:36.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.572+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 shutdown_connections 2024-03-29T11:23:36.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.572+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 >> 172.21.15.3:0/4100621174 conn(0x7f8fa0066a00 msgr2=0x7f8fa0134cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:36.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.573+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 shutdown_connections 2024-03-29T11:23:36.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.573+0000 7f8fa73c8640 1 -- 172.21.15.3:0/4100621174 wait complete. 2024-03-29T11:23:36.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:36.573+0000 7f8fa73c8640 1 librados: shutdown 2024-03-29T11:23:36.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:43.373 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:43.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.548+0000 7ffb3a0fb640 10 monclient: get_monmap_and_config 2024-03-29T11:23:43.532 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.548+0000 7ffb3a0fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.548+0000 7ffb3a0fb640 10 monclient: monmap: 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:43.549609+0000 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:43.549609+0000 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.549+0000 7ffb3a0fb640 1 Processor -- start 2024-03-29T11:23:43.533 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.549+0000 7ffb3a0fb640 1 -- start start 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.549+0000 7ffb3a0fb640 10 monclient: init 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb340a9910 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb340aab70 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb340a9cf0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb34146890 con 0x7ffb340a9910 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340abbd0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340aa8b0 con 0x7ffb340aab70 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:43.534 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.550+0000 7ffb3a0fb640 10 monclient(hunting): authenticate will time out at 1350.804818s 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.553+0000 7ffb32ffd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ffb340aab70 0x7ffb34146d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50810/0 (socket says 172.21.15.3:50810) 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.553+0000 7ffb32ffd640 1 -- 172.21.15.3:0/170314815 learned_addr learned my addr 172.21.15.3:0/170314815 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40229040 0 0) 0x7ffb340aa8b0 con 0x7ffb340aab70 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:43.535 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.554+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c003520 con 0x7ffb340aab70 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502061877 0 0) 0x7ffb1c003520 con 0x7ffb340aab70 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 >> v1:172.21.15.32:6789/0 conn(0x7ffb340a9cf0 legacy=0x7ffb34143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 >> v1:172.21.15.3:6789/0 conn(0x7ffb340a9910 legacy=0x7ffb340ae610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 monclient: found mon.noname-b 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:23:43.536 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb34143c50 con 0x7ffb340aab70 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb24001690 con 0x7ffb340aab70 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.555+0000 7ffb327fc640 10 monclient: dump: 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:43.537 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24001860 con 0x7ffb340aab70 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 1 -- 172.21.15.3:0/170314815 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb24001cb0 con 0x7ffb340aab70 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.556+0000 7ffb327fc640 10 monclient: dump: 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:43.538 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.557+0000 7ffb3a0fb640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.557+0000 7ffb3a0fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.557+0000 7ffb3a0fb640 4 set_mon_vals no callback set 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.558+0000 7ffb3a0fb640 10 monclient: shutdown 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.558+0000 7ffb3a0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:43.539 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.558+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/170314815 >> v1:172.21.15.3:6790/0 conn(0x7ffb340aab70 legacy=0x7ffb34146d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.560+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/170314815 shutdown_connections 2024-03-29T11:23:43.540 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.560+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/170314815 >> 172.21.15.3:0/170314815 conn(0x7ffb340b3c10 msgr2=0x7ffb340b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:43.541 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.561+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/170314815 shutdown_connections 2024-03-29T11:23:43.542 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.563+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/170314815 wait complete. 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.563+0000 7ffb3a0fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.563+0000 7ffb3a0fb640 10 monclient: monmap: 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:43.543 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.564+0000 7ffb3a0fb640 1 librados: starting msgr at 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.564+0000 7ffb3a0fb640 1 librados: starting objecter 2024-03-29T11:23:43.544 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.564+0000 7ffb3a0fb640 1 Processor -- start 2024-03-29T11:23:43.546 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.567+0000 7ffb3a0fb640 1 -- start start 2024-03-29T11:23:43.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.567+0000 7ffb3a0fb640 1 librados: setting wanted keys 2024-03-29T11:23:43.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.567+0000 7ffb3a0fb640 1 librados: calling monclient init 2024-03-29T11:23:43.547 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.567+0000 7ffb3a0fb640 10 monclient: init 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-b con 0x7ffb340a9910 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-a con 0x7ffb340a9cf0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient(hunting): picked mon.noname-c con 0x7ffb340aab70 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb341db730 con 0x7ffb340a9cf0 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb34141840 con 0x7ffb340a9910 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3413f100 con 0x7ffb340aab70 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:43.549 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.568+0000 7ffb3a0fb640 10 monclient(hunting): authenticate will time out at 1350.822321s 2024-03-29T11:23:43.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb32ffd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7ffb340a9cf0 0x7ffb341d88a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:35971/0 (socket says 172.21.15.3:35971) 2024-03-29T11:23:43.550 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb32ffd640 1 -- 172.21.15.3:0/2472723990 learned_addr learned my addr 172.21.15.3:0/2472723990 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:43.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 client.?.objecter ms_handle_connect 0x7ffb340a9cf0 2024-03-29T11:23:43.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:43.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369333184 0 0) 0x7ffb341db730 con 0x7ffb340a9cf0 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb20004eb0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 client.?.objecter ms_handle_connect 0x7ffb340aab70 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.571+0000 7ffb1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886922464 0 0) 0x7ffb3413f100 con 0x7ffb340aab70 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb341db730 con 0x7ffb340aab70 2024-03-29T11:23:43.553 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2340368369 0 0) 0x7ffb20004eb0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb200064f0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb24001b70 con 0x7ffb340a9cf0 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:43.554 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1088744848 0 0) 0x7ffb341db730 con 0x7ffb340aab70 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb20004eb0 con 0x7ffb340aab70 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb2c001bb0 con 0x7ffb340aab70 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:43.555 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.556 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:43.556 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:43.556 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:43.556 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 663009618 0 0) 0x7ffb200064f0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 >> v1:172.21.15.3:6790/0 conn(0x7ffb340aab70 legacy=0x7ffb3413fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.572+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 >> v1:172.21.15.32:6789/0 conn(0x7ffb340a9910 legacy=0x7ffb341dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:43.557 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 1 monclient: found mon.a 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb341dc2d0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 client.?.objecter ms_handle_connect 0x7ffb340a9910 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb1bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 10 monclient: _renew_subs 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb341d8fb0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:43.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 10 monclient: _renew_subs 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb341404d0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.573+0000 7ffb3a0fb640 1 librados: init done 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24004a90 con 0x7ffb340a9cf0 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb24004da0 con 0x7ffb340a9cf0 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb1bfff640 10 monclient: dump: 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:43.559 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.574+0000 7ffb31ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.575+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ffb2401ad70 con 0x7ffb340a9cf0 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.576+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7ffb24084a80 con 0x7ffb340a9cf0 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.576+0000 7ffb1bfff640 10 client.4542.objecter ms_dispatch 0x7ffb34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.576+0000 7ffb1bfff640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.576+0000 7ffb1bfff640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb1bfff640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb337fe640 10 monclient: get_auth_request con 0x7ffb2006df80 auth_method 0 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb1bfff640 10 client.4542.objecter ms_handle_connect 0x7ffb2006df80 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb3a0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:43.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb3a0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:43.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb3a0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:43.561 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.577+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb14005080 con 0x7ffb340a9cf0 2024-03-29T11:23:43.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.586+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb24050360 con 0x7ffb340a9cf0 2024-03-29T11:23:43.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.586+0000 7ffb1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:43.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.586+0000 7ffb1bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:43.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.924+0000 7ffb3a0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:43.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.924+0000 7ffb3a0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:43.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.924+0000 7ffb3a0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:23:43.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.924+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb14005330 con 0x7ffb340a9cf0 2024-03-29T11:23:43.906 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:43.906 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:43.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.925+0000 7ffb1bfff640 1 -- 172.21.15.3:0/2472723990 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7ffb24052290 con 0x7ffb340a9cf0 2024-03-29T11:23:43.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.925+0000 7ffb1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:43.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.925+0000 7ffb1bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.930+0000 7ffb3a0fb640 10 librados: watch_flush enter 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.930+0000 7ffb3a0fb640 10 librados: watch_flush exit 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.930+0000 7ffb3a0fb640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.930+0000 7ffb3a0fb640 10 client.4542.objecter successfully canceled tick 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 >> v1:172.21.15.3:6816/844689376 conn(0x7ffb2006df80 legacy=0x7ffb20070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 10 monclient: shutdown 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 >> v1:172.21.15.3:6789/0 conn(0x7ffb340a9cf0 legacy=0x7ffb341d88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 shutdown_connections 2024-03-29T11:23:43.911 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.931+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 >> 172.21.15.3:0/2472723990 conn(0x7ffb340b3c10 msgr2=0x7ffb34069b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:43.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.933+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 shutdown_connections 2024-03-29T11:23:43.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.933+0000 7ffb3a0fb640 1 -- 172.21.15.3:0/2472723990 wait complete. 2024-03-29T11:23:43.914 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:43.935+0000 7ffb3a0fb640 1 librados: shutdown 2024-03-29T11:23:43.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:50.740 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:50.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.909+0000 7f580e109640 10 monclient: get_monmap_and_config 2024-03-29T11:23:50.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.909+0000 7f580e109640 10 monclient: build_initial_monmap 2024-03-29T11:23:50.890 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.909+0000 7f580e109640 10 monclient: monmap: 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:50.910141+0000 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:50.910141+0000 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:50.891 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.910+0000 7f580e109640 1 Processor -- start 2024-03-29T11:23:50.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 1 -- start start 2024-03-29T11:23:50.892 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient: init 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient(hunting): picked mon.noname-c con 0x7f5800029eb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient(hunting): picked mon.noname-b con 0x7f580002b0a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 10 monclient(hunting): picked mon.noname-a con 0x7f58000c9570 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.912+0000 7f580e109640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580002a290 con 0x7f58000c9570 2024-03-29T11:23:50.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.913+0000 7f580e109640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58000331c0 con 0x7f5800029eb0 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.913+0000 7f580e109640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580002c100 con 0x7f580002b0a0 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.913+0000 7f580e109640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.913+0000 7f580e109640 10 monclient(hunting): authenticate will time out at 1358.167013s 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.914+0000 7f580c906640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f580002b0a0 0x7f58000c8e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:33242/0 (socket says 172.21.15.3:33242) 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.914+0000 7f580c906640 1 -- 172.21.15.3:0/2989198561 learned_addr learned my addr 172.21.15.3:0/2989198561 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:50.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.915+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521516733 0 0) 0x7f580002c100 con 0x7f580002b0a0 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57f4003520 con 0x7f580002b0a0 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193917545 0 0) 0x7f580002a290 con 0x7f58000c9570 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:23:50.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580002c100 con 0x7f58000c9570 2024-03-29T11:23:50.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 442918964 0 0) 0x7f57f4003520 con 0x7f580002b0a0 2024-03-29T11:23:50.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.916+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 >> v1:172.21.15.32:6789/0 conn(0x7f5800029eb0 legacy=0x7f58000cf120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:50.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 >> v1:172.21.15.3:6789/0 conn(0x7f58000c9570 legacy=0x7f58000d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:50.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:50.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 1 monclient: found mon.noname-b 2024-03-29T11:23:50.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580002a440 con 0x7f580002b0a0 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: _finish_auth 0 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f580804f730 con 0x7f580002b0a0 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: dump: 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.899 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f580e109640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f580e109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5808075f80 con 0x7f580002b0a0 2024-03-29T11:23:50.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f57fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f580e109640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.918+0000 7f580e109640 4 set_mon_vals no callback set 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f57fffff640 1 -- 172.21.15.3:0/2989198561 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f580804f080 con 0x7f580002b0a0 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f57fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f57fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f57fffff640 10 monclient: dump: 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.901 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f580e109640 10 monclient: shutdown 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f580e109640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f580e109640 1 -- 172.21.15.3:0/2989198561 >> v1:172.21.15.3:6790/0 conn(0x7f580002b0a0 legacy=0x7f58000c8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f580e109640 1 -- 172.21.15.3:0/2989198561 shutdown_connections 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.919+0000 7f580e109640 1 -- 172.21.15.3:0/2989198561 >> 172.21.15.3:0/2989198561 conn(0x7f58000058f0 msgr2=0x7f5800005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.920+0000 7f580e109640 1 -- 172.21.15.3:0/2989198561 shutdown_connections 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.920+0000 7f580e109640 1 -- 172.21.15.3:0/2989198561 wait complete. 2024-03-29T11:23:50.902 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.921+0000 7f580e109640 10 monclient: build_initial_monmap 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.921+0000 7f580e109640 10 monclient: monmap: 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:50.903 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.921+0000 7f580e109640 1 librados: starting msgr at 2024-03-29T11:23:50.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.921+0000 7f580e109640 1 librados: starting objecter 2024-03-29T11:23:50.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.921+0000 7f580e109640 1 Processor -- start 2024-03-29T11:23:50.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.923+0000 7f580e109640 1 -- start start 2024-03-29T11:23:50.906 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.923+0000 7f580e109640 1 librados: setting wanted keys 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.923+0000 7f580e109640 1 librados: calling monclient init 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.923+0000 7f580e109640 10 monclient: init 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient(hunting): picked mon.noname-a con 0x7f5800029eb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient(hunting): picked mon.noname-b con 0x7f580002b0a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient(hunting): picked mon.noname-c con 0x7f58000c9570 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:50.907 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f580002e330 con 0x7f5800029eb0 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58000ccaa0 con 0x7f580002b0a0 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5800161720 con 0x7f58000c9570 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580e109640 10 monclient(hunting): authenticate will time out at 1358.178439s 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580d908640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f58000c9570 0x7f580016b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:36384/0 (socket says 172.21.15.3:36384) 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.924+0000 7f580d908640 1 -- 172.21.15.3:0/2600302423 learned_addr learned my addr 172.21.15.3:0/2600302423 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:50.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 client.?.objecter ms_handle_connect 0x7f58000c9570 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904676844 0 0) 0x7f5800161720 con 0x7f58000c9570 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.925+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57ec004eb0 con 0x7f58000c9570 2024-03-29T11:23:50.909 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381443174 0 0) 0x7f57ec004eb0 con 0x7f58000c9570 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57ec005300 con 0x7f58000c9570 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f57f80018f0 con 0x7f58000c9570 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 10 monclient(hunting): dump: 2024-03-29T11:23:50.910 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:50.911 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:50.912 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:50.912 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:50.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1179076353 0 0) 0x7f57ec005300 con 0x7f58000c9570 2024-03-29T11:23:50.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.926+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 >> v1:172.21.15.32:6789/0 conn(0x7f580002b0a0 legacy=0x7f5800161010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:50.915 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.927+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 >> v1:172.21.15.3:6789/0 conn(0x7f5800029eb0 legacy=0x7f58000cc390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 1 monclient: found mon.c 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58001618b0 con 0x7f58000c9570 2024-03-29T11:23:50.916 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 client.?.objecter ms_handle_connect 0x7f5800029eb0 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 10 monclient: _renew_subs 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58000c99a0 con 0x7f58000c9570 2024-03-29T11:23:50.917 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 10 monclient: _renew_subs 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f580002cce0 con 0x7f58000c9570 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57f80035f0 con 0x7f58000c9570 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f580e109640 1 librados: init done 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f57f80018f0 con 0x7f58000c9570 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:50.918 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57fdffb640 10 monclient: dump: 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.931+0000 7f57feffd640 4 set_mon_vals no callback set 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.933+0000 7f580e109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.933+0000 7f580e109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.933+0000 7f580e109640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:50.919 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.933+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f580002ad00 con 0x7f58000c9570 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.934+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f57f8003760 con 0x7f58000c9570 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.935+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f57f8085630 con 0x7f58000c9570 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.935+0000 7f57fdffb640 10 client.4577.objecter ms_dispatch 0x7f5808000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.935+0000 7f57fdffb640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.935+0000 7f57fdffb640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.935+0000 7f57fdffb640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.937+0000 7f580d107640 10 monclient: get_auth_request con 0x7f57ec06df50 auth_method 0 2024-03-29T11:23:50.920 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.937+0000 7f57fdffb640 10 client.4577.objecter ms_handle_connect 0x7f57ec06df50 2024-03-29T11:23:50.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.943+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57f804fd10 con 0x7f58000c9570 2024-03-29T11:23:50.925 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.944+0000 7f57fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:50.926 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:50.944+0000 7f57fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:51.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.272+0000 7f580e109640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:51.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.272+0000 7f580e109640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:51.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.272+0000 7f580e109640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:23:51.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.272+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5800160950 con 0x7f58000c9570 2024-03-29T11:23:51.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.274+0000 7f57fdffb640 1 -- 172.21.15.3:0/2600302423 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f5800160950 con 0x7f58000c9570 2024-03-29T11:23:51.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.274+0000 7f57fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:51.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.274+0000 7f57fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:51.255 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:51.255 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:51.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 10 librados: watch_flush enter 2024-03-29T11:23:51.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 10 librados: watch_flush exit 2024-03-29T11:23:51.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:23:51.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:23:51.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 >> v1:172.21.15.3:6816/844689376 conn(0x7f57ec06df50 legacy=0x7f57ec070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:51.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 10 monclient: shutdown 2024-03-29T11:23:51.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:51.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.278+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 >> v1:172.21.15.3:6790/0 conn(0x7f58000c9570 legacy=0x7f580016b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:51.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.279+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 shutdown_connections 2024-03-29T11:23:51.260 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.279+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 >> 172.21.15.3:0/2600302423 conn(0x7f58000058f0 msgr2=0x7f58000043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:51.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.280+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 shutdown_connections 2024-03-29T11:23:51.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.280+0000 7f580e109640 1 -- 172.21.15.3:0/2600302423 wait complete. 2024-03-29T11:23:51.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:51.280+0000 7f580e109640 1 librados: shutdown 2024-03-29T11:23:51.276 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:58.082 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.253+0000 7f69216ba640 10 monclient: get_monmap_and_config 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.253+0000 7f69216ba640 10 monclient: build_initial_monmap 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.253+0000 7f69216ba640 10 monclient: monmap: 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:23:58.255450+0000 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:23:58.255450+0000 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.235 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:58.236 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:23:58.236 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:23:58.236 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.237 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.254+0000 7f69216ba640 1 Processor -- start 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.254+0000 7f69216ba640 1 -- start start 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.254+0000 7f69216ba640 10 monclient: init 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-c con 0x7f691c0a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-b con 0x7f691c0aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-a con 0x7f691c146f80 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:58.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0aed20 con 0x7f691c146f80 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c0aa8b0 con 0x7f691c0a98a0 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c146a60 con 0x7f691c0aab00 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.255+0000 7f69216ba640 10 monclient(hunting): authenticate will time out at 1365.510527s 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.258+0000 7f691affd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f691c0a98a0 0x7f691c0ae610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:44751/0 (socket says 172.21.15.3:44751) 2024-03-29T11:23:58.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.258+0000 7f691affd640 1 -- 172.21.15.3:0/3305272433 learned_addr learned my addr 172.21.15.3:0/3305272433 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:58.244 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672504699 0 0) 0x7f691c0aa8b0 con 0x7f691c0a98a0 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6904003520 con 0x7f691c0a98a0 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976495712 0 0) 0x7f691c0aed20 con 0x7f691c146f80 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.261+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c0aa8b0 con 0x7f691c146f80 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816410306 0 0) 0x7f691c146a60 con 0x7f691c0aab00 2024-03-29T11:23:58.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c0aed20 con 0x7f691c0aab00 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2216893113 0 0) 0x7f6904003520 con 0x7f691c0a98a0 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 >> v1:172.21.15.3:6790/0 conn(0x7f691c0aab00 legacy=0x7f691c1437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 >> v1:172.21.15.3:6789/0 conn(0x7f691c146f80 legacy=0x7f691c151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c143ef0 con 0x7f691c0a98a0 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:58.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6910001690 con 0x7f691c0a98a0 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: dump: 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:58.247 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f69216ba640 5 monclient: authenticate success, global_id 4558 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f69216ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69100017f0 con 0x7f691c0a98a0 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f6919ffb640 1 -- 172.21.15.3:0/3305272433 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6910001c40 con 0x7f691c0a98a0 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f69216ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.262+0000 7f69216ba640 4 set_mon_vals no callback set 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f6919ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f6919ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f6919ffb640 10 monclient: dump: 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.248 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f69216ba640 10 monclient: shutdown 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f69216ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:58.249 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.263+0000 7f69216ba640 1 -- 172.21.15.3:0/3305272433 >> v1:172.21.15.32:6789/0 conn(0x7f691c0a98a0 legacy=0x7f691c0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.250 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.266+0000 7f69216ba640 1 -- 172.21.15.3:0/3305272433 shutdown_connections 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.266+0000 7f69216ba640 1 -- 172.21.15.3:0/3305272433 >> 172.21.15.3:0/3305272433 conn(0x7f691c0b3c10 msgr2=0x7f691c0b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.266+0000 7f69216ba640 1 -- 172.21.15.3:0/3305272433 shutdown_connections 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.266+0000 7f69216ba640 1 -- 172.21.15.3:0/3305272433 wait complete. 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.267+0000 7f69216ba640 10 monclient: build_initial_monmap 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.267+0000 7f69216ba640 10 monclient: monmap: 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:58.251 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.267+0000 7f69216ba640 1 librados: starting msgr at 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.267+0000 7f69216ba640 1 librados: starting objecter 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.268+0000 7f69216ba640 1 Processor -- start 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.268+0000 7f69216ba640 1 -- start start 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.268+0000 7f69216ba640 1 librados: setting wanted keys 2024-03-29T11:23:58.252 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.268+0000 7f69216ba640 1 librados: calling monclient init 2024-03-29T11:23:58.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.268+0000 7f69216ba640 10 monclient: init 2024-03-29T11:23:58.253 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.269+0000 7f69216ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.269+0000 7f69216ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.269+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-b con 0x7f691c0a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.269+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-a con 0x7f691c0aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 10 monclient(hunting): picked mon.noname-c con 0x7f691c146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c1489e0 con 0x7f691c0aab00 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c149f20 con 0x7f691c0a98a0 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f691c147850 con 0x7f691c146f80 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:58.254 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.271+0000 7f69216ba640 10 monclient(hunting): authenticate will time out at 1365.526905s 2024-03-29T11:23:58.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.275+0000 7f691b7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f691c146f80 0x7f691c1df030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40071/0 (socket says 172.21.15.3:40071) 2024-03-29T11:23:58.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.275+0000 7f691b7fe640 1 -- 172.21.15.3:0/2985193661 learned_addr learned my addr 172.21.15.3:0/2985193661 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:23:58.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.275+0000 7f691a7fc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f691c0aab00 0x7f691c1de920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:52879/0 (socket says 172.21.15.3:52879) 2024-03-29T11:23:58.257 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.275+0000 7f69037fe640 10 client.?.objecter ms_handle_connect 0x7f691c0aab00 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 client.?.objecter ms_handle_connect 0x7f691c0a98a0 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833174734 0 0) 0x7f691c1489e0 con 0x7f691c0aab00 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6908004eb0 con 0x7f691c0aab00 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260727900 0 0) 0x7f691c149f20 con 0x7f691c0a98a0 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:58.258 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:23:58.259 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.276+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f691c1489e0 con 0x7f691c0a98a0 2024-03-29T11:23:58.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131557391 0 0) 0x7f691c1489e0 con 0x7f691c0a98a0 2024-03-29T11:23:58.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f691c149f20 con 0x7f691c0a98a0 2024-03-29T11:23:58.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6910001b00 con 0x7f691c0a98a0 2024-03-29T11:23:58.261 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 10 monclient(hunting): dump: 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.262 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.277+0000 7f69037fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.278+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2718014567 0 0) 0x7f691c149f20 con 0x7f691c0a98a0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 >> v1:172.21.15.3:6790/0 conn(0x7f691c146f80 legacy=0x7f691c1df030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 >> v1:172.21.15.3:6789/0 conn(0x7f691c0aab00 legacy=0x7f691c1de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 monclient: found mon.b 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691c137350 con 0x7f691c0a98a0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69216ba640 5 monclient: authenticate success, global_id 4561 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69216ba640 10 monclient: _renew_subs 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69216ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.263 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c136f40 con 0x7f691c0a98a0 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69216ba640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6910001c60 con 0x7f691c0a98a0 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f6910004d90 con 0x7f691c0a98a0 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.279+0000 7f69037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69037fe640 10 monclient: dump: 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:23:58.264 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f6918ff9640 4 set_mon_vals no callback set 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69216ba640 10 monclient: _renew_subs 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69216ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c1484a0 con 0x7f691c0a98a0 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.280+0000 7f69216ba640 1 librados: init done 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.282+0000 7f69216ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:58.265 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.282+0000 7f69216ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:58.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.282+0000 7f69216ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.266 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.282+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c1487f0 con 0x7f691c0a98a0 2024-03-29T11:23:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.285+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f691001ad60 con 0x7f691c0a98a0 2024-03-29T11:23:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f6910084670 con 0x7f691c0a98a0 2024-03-29T11:23:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f69037fe640 10 client.4561.objecter ms_dispatch 0x7f691c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:58.267 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f69037fe640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:58.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f69037fe640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:58.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f69037fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-29T11:23:58.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.286+0000 7f691a7fc640 10 monclient: get_auth_request con 0x7f690806df50 auth_method 0 2024-03-29T11:23:58.268 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.287+0000 7f69037fe640 10 client.4561.objecter ms_handle_connect 0x7f690806df50 2024-03-29T11:23:58.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.291+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f691004ff50 con 0x7f691c0a98a0 2024-03-29T11:23:58.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.291+0000 7f69037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:58.272 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.291+0000 7f69037fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:58.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.636+0000 7f69216ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:58.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.636+0000 7f69216ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:58.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.636+0000 7f69216ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:23:58.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.636+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f691c1487f0 con 0x7f691c0a98a0 2024-03-29T11:23:58.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.639+0000 7f69037fe640 1 -- 172.21.15.3:0/2985193661 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f691c1487f0 con 0x7f691c0a98a0 2024-03-29T11:23:58.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.639+0000 7f69037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:58.621 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.639+0000 7f69037fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:58.621 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:23:58.621 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:58.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 10 librados: watch_flush enter 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 10 librados: watch_flush exit 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 10 client.4561.objecter successfully canceled tick 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 >> v1:172.21.15.3:6816/844689376 conn(0x7f690806df50 legacy=0x7f6908070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 10 monclient: shutdown 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 >> v1:172.21.15.32:6789/0 conn(0x7f691c0a98a0 legacy=0x7f691c136830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 shutdown_connections 2024-03-29T11:23:58.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.643+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 >> 172.21.15.3:0/2985193661 conn(0x7f691c0b3c10 msgr2=0x7f691c0ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:58.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.645+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 shutdown_connections 2024-03-29T11:23:58.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.646+0000 7f69216ba640 1 -- 172.21.15.3:0/2985193661 wait complete. 2024-03-29T11:23:58.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:23:58.647+0000 7f69216ba640 1 librados: shutdown 2024-03-29T11:23:58.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:05.452 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:05.606 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.620+0000 7f515a00f640 10 monclient: get_monmap_and_config 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.620+0000 7f515a00f640 10 monclient: build_initial_monmap 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.620+0000 7f515a00f640 10 monclient: monmap: 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:05.621746+0000 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:05.621746+0000 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.622+0000 7f515a00f640 1 Processor -- start 2024-03-29T11:24:05.607 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.622+0000 7f515a00f640 1 -- start start 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.622+0000 7f515a00f640 10 monclient: init 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-c con 0x7f515414f8d0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-a con 0x7f51540aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-b con 0x7f51540a9d40 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515414f3b0 con 0x7f51540aab00 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515408ba80 con 0x7f515414f8d0 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515408bc10 con 0x7f51540a9d40 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f515a00f640 10 monclient(hunting): authenticate will time out at 1372.877313s 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f5153fff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f51540a9d40 0x7f515408b370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:54340/0 (socket says 172.21.15.3:54340) 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.623+0000 7f5153fff640 1 -- 172.21.15.3:0/957276633 learned_addr learned my addr 172.21.15.3:0/957276633 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:05.608 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183587591 0 0) 0x7f515408bc10 con 0x7f51540a9d40 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.624+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f513c003520 con 0x7f51540a9d40 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.625+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693793263 0 0) 0x7f513c003520 con 0x7f51540a9d40 2024-03-29T11:24:05.609 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.625+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 >> v1:172.21.15.32:6789/0 conn(0x7f515414f8d0 legacy=0x7f51540ae610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.626+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 >> v1:172.21.15.3:6789/0 conn(0x7f51540aab00 legacy=0x7f51540a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 1 monclient: found mon.noname-b 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515408bda0 con 0x7f51540a9d40 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5144001690 con 0x7f51540a9d40 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:05.610 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f51527fc640 10 monclient: dump: 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f515a00f640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.628+0000 7f515a00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:05.611 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5144001860 con 0x7f51540a9d40 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 1 -- 172.21.15.3:0/957276633 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5144001cb0 con 0x7f51540a9d40 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f51527fc640 10 monclient: dump: 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.612 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:05.613 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:05.613 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:05.613 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f515a00f640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:05.613 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f515a00f640 4 set_mon_vals no callback set 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f515a00f640 10 monclient: shutdown 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.629+0000 7f515a00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.630+0000 7f515a00f640 1 -- 172.21.15.3:0/957276633 >> v1:172.21.15.3:6790/0 conn(0x7f51540a9d40 legacy=0x7f515408b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.630+0000 7f515a00f640 1 -- 172.21.15.3:0/957276633 shutdown_connections 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.630+0000 7f515a00f640 1 -- 172.21.15.3:0/957276633 >> 172.21.15.3:0/957276633 conn(0x7f51540b3c10 msgr2=0x7f51540b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:05.616 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.630+0000 7f515a00f640 1 -- 172.21.15.3:0/957276633 shutdown_connections 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.630+0000 7f515a00f640 1 -- 172.21.15.3:0/957276633 wait complete. 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.631+0000 7f515a00f640 10 monclient: build_initial_monmap 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.631+0000 7f515a00f640 10 monclient: monmap: 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:05.617 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 librados: starting msgr at 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 librados: starting objecter 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 Processor -- start 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 -- start start 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 librados: setting wanted keys 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 1 librados: calling monclient init 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.632+0000 7f515a00f640 10 monclient: init 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-a con 0x7f51540a9d40 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-b con 0x7f51540aab00 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient(hunting): picked mon.noname-c con 0x7f515414f8d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515413de30 con 0x7f51540a9d40 2024-03-29T11:24:05.618 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515414cca0 con 0x7f51540aab00 2024-03-29T11:24:05.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515414d8d0 con 0x7f515414f8d0 2024-03-29T11:24:05.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:05.619 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.633+0000 7f515a00f640 10 monclient(hunting): authenticate will time out at 1372.887535s 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f5152ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f51540aab00 0x7f51541f0400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:54781/0 (socket says 172.21.15.3:54781) 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f5152ffd640 1 -- 172.21.15.3:0/1996547701 learned_addr learned my addr 172.21.15.3:0/1996547701 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 client.?.objecter ms_handle_connect 0x7f51540aab00 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203747823 0 0) 0x7f515414cca0 con 0x7f51540aab00 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.637+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5140004eb0 con 0x7f51540aab00 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120406930 0 0) 0x7f5140004eb0 con 0x7f51540aab00 2024-03-29T11:24:05.622 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f515414cca0 con 0x7f51540aab00 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5148001990 con 0x7f51540aab00 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 10 monclient(hunting): dump: 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.623 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.638+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1789137211 0 0) 0x7f515414cca0 con 0x7f51540aab00 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 >> v1:172.21.15.3:6790/0 conn(0x7f515414f8d0 legacy=0x7f515414d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 >> v1:172.21.15.3:6789/0 conn(0x7f51540a9d40 legacy=0x7f51541efcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 1 monclient: found mon.b 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515414da60 con 0x7f51540aab00 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 10 monclient: _finish_auth 0 2024-03-29T11:24:05.624 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f513bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 5 monclient: authenticate success, global_id 4564 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 10 monclient: _renew_subs 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515414dd10 con 0x7f51540aab00 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 10 monclient: _renew_subs 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51541f81b0 con 0x7f51540aab00 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.639+0000 7f515a00f640 1 librados: init done 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.640+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5148001cb0 con 0x7f51540aab00 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.640+0000 7f513bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:05.625 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.640+0000 7f5151ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f515a00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f5148003a60 con 0x7f51540aab00 2024-03-29T11:24:05.626 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f515a00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f515a00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5134005080 con 0x7f51540aab00 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f513bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f513bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f513bfff640 10 monclient: dump: 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:05.627 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.641+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f5148003c80 con 0x7f51540aab00 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.642+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f5148084af0 con 0x7f51540aab00 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.642+0000 7f513bfff640 10 client.4564.objecter ms_dispatch 0x7f5154000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.642+0000 7f513bfff640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.642+0000 7f513bfff640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.642+0000 7f513bfff640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.644+0000 7f51537fe640 10 monclient: get_auth_request con 0x7f514006dcf0 auth_method 0 2024-03-29T11:24:05.628 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.645+0000 7f513bfff640 10 client.4564.objecter ms_handle_connect 0x7f514006dcf0 2024-03-29T11:24:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.649+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5148050450 con 0x7f51540aab00 2024-03-29T11:24:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.649+0000 7f513bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:05.632 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.649+0000 7f513bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:05.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.994+0000 7f515a00f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:05.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.994+0000 7f515a00f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:05.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.994+0000 7f515a00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:05.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.994+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5134005330 con 0x7f51540aab00 2024-03-29T11:24:05.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.996+0000 7f513bfff640 1 -- 172.21.15.3:0/1996547701 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f5148052380 con 0x7f51540aab00 2024-03-29T11:24:05.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.996+0000 7f513bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:05.977 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.996+0000 7f513bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:05.977 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:05.977 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:05.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 10 librados: watch_flush enter 2024-03-29T11:24:05.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 10 librados: watch_flush exit 2024-03-29T11:24:05.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-29T11:24:05.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 10 client.4564.objecter successfully canceled tick 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 >> v1:172.21.15.3:6816/844689376 conn(0x7f514006dcf0 legacy=0x7f51400701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 10 monclient: shutdown 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:05.999+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 >> v1:172.21.15.32:6789/0 conn(0x7f51540aab00 legacy=0x7f51541f0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:06.000+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 shutdown_connections 2024-03-29T11:24:05.982 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:06.000+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 >> 172.21.15.3:0/1996547701 conn(0x7f51540b3c10 msgr2=0x7f515408b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:05.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:06.000+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 shutdown_connections 2024-03-29T11:24:05.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:06.000+0000 7f515a00f640 1 -- 172.21.15.3:0/1996547701 wait complete. 2024-03-29T11:24:05.983 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:06.000+0000 7f515a00f640 1 librados: shutdown 2024-03-29T11:24:05.998 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:12.805 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.971+0000 7fa35a8cf640 10 monclient: get_monmap_and_config 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.971+0000 7fa35a8cf640 10 monclient: build_initial_monmap 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.971+0000 7fa35a8cf640 10 monclient: monmap: 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:12.972022+0000 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:12.972022+0000 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:12.952 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 1 Processor -- start 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 1 -- start start 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 10 monclient: init 2024-03-29T11:24:12.953 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:12.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:12.954 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.972+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa3540b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:12.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3540b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:12.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3540b3560 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:12.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540aab60 con 0x7fa3540b3560 2024-03-29T11:24:12.955 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540aae40 con 0x7fa3540b2cb0 2024-03-29T11:24:12.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa3537fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa3540b2cb0 0x7fa354142190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:59006/0 (socket says 172.21.15.3:59006) 2024-03-29T11:24:12.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa3537fe640 1 -- 172.21.15.3:0/1697629184 learned_addr learned my addr 172.21.15.3:0/1697629184 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:12.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3540ab120 con 0x7fa3540b28d0 2024-03-29T11:24:12.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:12.956 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.974+0000 7fa35a8cf640 10 monclient(hunting): authenticate will time out at 1380.228327s 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805539959 0 0) 0x7fa3540aae40 con 0x7fa3540b2cb0 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:24:12.957 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.975+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c003520 con 0x7fa3540b2cb0 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235511918 0 0) 0x7fa3540ab120 con 0x7fa3540b28d0 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.976+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3540aae40 con 0x7fa3540b28d0 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 13150565 0 0) 0x7fa3540aae40 con 0x7fa3540b28d0 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 >> v1:172.21.15.32:6789/0 conn(0x7fa3540b2cb0 legacy=0x7fa354142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 >> v1:172.21.15.3:6789/0 conn(0x7fa3540b3560 legacy=0x7fa3541428a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:12.959 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 monclient: found mon.noname-b 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35413c380 con 0x7fa3540b28d0 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa344001670 con 0x7fa3540b28d0 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: dump: 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:12.960 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa35a8cf640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa35a8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa344001850 con 0x7fa3540b28d0 2024-03-29T11:24:12.961 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 1 -- 172.21.15.3:0/1697629184 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa344001ca0 con 0x7fa3540b28d0 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa352ffd640 10 monclient: dump: 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:12.962 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:12.963 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa35a8cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:12.963 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.977+0000 7fa35a8cf640 4 set_mon_vals no callback set 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.979+0000 7fa35a8cf640 10 monclient: shutdown 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.979+0000 7fa35a8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.979+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 >> v1:172.21.15.3:6790/0 conn(0x7fa3540b28d0 legacy=0x7fa35413bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.981+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 shutdown_connections 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.981+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 >> 172.21.15.3:0/1697629184 conn(0x7fa354066a00 msgr2=0x7fa354066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.981+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 shutdown_connections 2024-03-29T11:24:12.964 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.981+0000 7fa35a8cf640 1 -- 172.21.15.3:0/1697629184 wait complete. 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.982+0000 7fa35a8cf640 10 monclient: build_initial_monmap 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.982+0000 7fa35a8cf640 10 monclient: monmap: 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:12.965 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.982+0000 7fa35a8cf640 1 librados: starting msgr at 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.982+0000 7fa35a8cf640 1 librados: starting objecter 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.983+0000 7fa35a8cf640 1 Processor -- start 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.983+0000 7fa35a8cf640 1 -- start start 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.983+0000 7fa35a8cf640 1 librados: setting wanted keys 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.983+0000 7fa35a8cf640 1 librados: calling monclient init 2024-03-29T11:24:12.966 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.983+0000 7fa35a8cf640 10 monclient: init 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-a con 0x7fa3540b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-b con 0x7fa35413eee0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient(hunting): picked mon.noname-c con 0x7fa3541df6e0 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35412ed60 con 0x7fa3540b28d0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3541da560 con 0x7fa35413eee0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa35413ea50 con 0x7fa3541df6e0 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:12.967 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.984+0000 7fa35a8cf640 10 monclient(hunting): authenticate will time out at 1380.238823s 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa353fff640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa3540b28d0 0x7fa35413e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42323/0 (socket says 172.21.15.3:42323) 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa353fff640 1 -- 172.21.15.3:0/70438981 learned_addr learned my addr 172.21.15.3:0/70438981 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 client.?.objecter ms_handle_connect 0x7fa3540b28d0 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1567189845 0 0) 0x7fa35412ed60 con 0x7fa3540b28d0 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:12.969 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa340004eb0 con 0x7fa3540b28d0 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3598942907 0 0) 0x7fa340004eb0 con 0x7fa3540b28d0 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa35412ed60 con 0x7fa3540b28d0 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa3440016f0 con 0x7fa3540b28d0 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.970 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546957828 0 0) 0x7fa35412ed60 con 0x7fa3540b28d0 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.987+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 >> v1:172.21.15.3:6790/0 conn(0x7fa3541df6e0 legacy=0x7fa3541e1ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 >> v1:172.21.15.32:6789/0 conn(0x7fa35413eee0 legacy=0x7fa3541daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 1 monclient: found mon.a 2024-03-29T11:24:12.971 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3541db150 con 0x7fa3540b28d0 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 10 monclient: _renew_subs 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:24:12.972 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3541e31f0 con 0x7fa3540b28d0 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 10 client.4560.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 10 monclient: _renew_subs 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3541e3560 con 0x7fa3540b28d0 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa35a8cf640 1 librados: init done 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa344003940 con 0x7fa3540b28d0 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa344003cd0 con 0x7fa3540b28d0 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:12.973 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa350ff9640 10 monclient: dump: 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:12.974 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.988+0000 7fa3527fc640 4 set_mon_vals no callback set 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.990+0000 7fa35a8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.990+0000 7fa35a8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.990+0000 7fa35a8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.990+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa35406a400 con 0x7fa3540b28d0 2024-03-29T11:24:12.975 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.990+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fa344003e30 con 0x7fa3540b28d0 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fa344084d40 con 0x7fa3540b28d0 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa350ff9640 10 client.4560.objecter ms_dispatch 0x7fa354000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa350ff9640 3 client.4560.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa350ff9640 3 client.4560.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa350ff9640 20 client.4560.objecter dump_active .. 0 homeless 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.991+0000 7fa3537fe640 10 monclient: get_auth_request con 0x7fa34006dc40 auth_method 0 2024-03-29T11:24:12.976 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:12.992+0000 7fa350ff9640 10 client.4560.objecter ms_handle_connect 0x7fa34006dc40 2024-03-29T11:24:12.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.000+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa344050620 con 0x7fa3540b28d0 2024-03-29T11:24:12.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.000+0000 7fa350ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:12.981 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.000+0000 7fa350ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:13.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.328+0000 7fa35a8cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:13.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.328+0000 7fa35a8cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:13.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.328+0000 7fa35a8cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:24:13.309 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.328+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa354072290 con 0x7fa3540b28d0 2024-03-29T11:24:13.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.330+0000 7fa350ff9640 1 -- 172.21.15.3:0/70438981 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fa344052550 con 0x7fa3540b28d0 2024-03-29T11:24:13.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.330+0000 7fa350ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:13.311 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.330+0000 7fa350ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:13.311 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:13.311 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:13.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.334+0000 7fa35a8cf640 10 librados: watch_flush enter 2024-03-29T11:24:13.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.334+0000 7fa35a8cf640 10 librados: watch_flush exit 2024-03-29T11:24:13.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.334+0000 7fa35a8cf640 20 client.4560.objecter shutdown clearing up homeless session... 2024-03-29T11:24:13.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.334+0000 7fa35a8cf640 10 client.4560.objecter successfully canceled tick 2024-03-29T11:24:13.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.335+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 >> v1:172.21.15.3:6816/844689376 conn(0x7fa34006dc40 legacy=0x7fa340070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:13.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.335+0000 7fa35a8cf640 10 monclient: shutdown 2024-03-29T11:24:13.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.335+0000 7fa35a8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.335+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 >> v1:172.21.15.3:6789/0 conn(0x7fa3540b28d0 legacy=0x7fa35413e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.336+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 shutdown_connections 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.336+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 >> 172.21.15.3:0/70438981 conn(0x7fa354066a00 msgr2=0x7fa35413fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.336+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 shutdown_connections 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.336+0000 7fa35a8cf640 1 -- 172.21.15.3:0/70438981 wait complete. 2024-03-29T11:24:13.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:13.336+0000 7fa35a8cf640 1 librados: shutdown 2024-03-29T11:24:13.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:20.144 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.315+0000 7f88d0ee4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.315+0000 7f88d0ee4640 10 monclient: build_initial_monmap 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.315+0000 7f88d0ee4640 10 monclient: monmap: 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:20.316744+0000 2024-03-29T11:24:20.297 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:20.316744+0000 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:20.298 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.316+0000 7f88d0ee4640 1 Processor -- start 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.316+0000 7f88d0ee4640 1 -- start start 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.316+0000 7f88d0ee4640 10 monclient: init 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:20.300 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f88cc0a98a0 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f88cc0aab00 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f88cc146f80 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc0aed20 con 0x7f88cc0a98a0 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc0aa8b0 con 0x7f88cc146f80 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc146a60 con 0x7f88cc0aab00 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88d0ee4640 10 monclient(hunting): authenticate will time out at 1387.572343s 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88ca59c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88cc0a98a0 0x7f88cc0ae610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:45594/0 (socket says 172.21.15.3:45594) 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.318+0000 7f88ca59c640 1 -- 172.21.15.3:0/1027005479 learned_addr learned my addr 172.21.15.3:0/1027005479 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543988369 0 0) 0x7f88cc146a60 con 0x7f88cc0aab00 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 10 monclient(hunting): _negotiate 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:24:20.301 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b4003520 con 0x7f88cc0aab00 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.319+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2999171191 0 0) 0x7f88b4003520 con 0x7f88cc0aab00 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 >> v1:172.21.15.32:6789/0 conn(0x7f88cc146f80 legacy=0x7f88cc151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 >> v1:172.21.15.3:6789/0 conn(0x7f88cc0a98a0 legacy=0x7f88cc0ae610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: _finish_hunting 0 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 monclient: found mon.noname-b 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc143ef0 con 0x7f88cc0aab00 2024-03-29T11:24:20.304 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: _finish_auth 0 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: _check_auth_tickets 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88d0ee4640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88d0ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f88c0001690 con 0x7f88cc0aab00 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: dump: 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.305 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88d0ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88c00017f0 con 0x7f88cc0aab00 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88d0ee4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88d0ee4640 4 set_mon_vals no callback set 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 1 -- 172.21.15.3:0/1027005479 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f88c0001c40 con 0x7f88cc0aab00 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.320+0000 7f88c959a640 10 monclient: dump: 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.306 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.321+0000 7f88d0ee4640 10 monclient: shutdown 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.321+0000 7f88d0ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.321+0000 7f88d0ee4640 1 -- 172.21.15.3:0/1027005479 >> v1:172.21.15.3:6790/0 conn(0x7f88cc0aab00 legacy=0x7f88cc1437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.321+0000 7f88d0ee4640 1 -- 172.21.15.3:0/1027005479 shutdown_connections 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.321+0000 7f88d0ee4640 1 -- 172.21.15.3:0/1027005479 >> 172.21.15.3:0/1027005479 conn(0x7f88cc0b3c10 msgr2=0x7f88cc0b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.322+0000 7f88d0ee4640 1 -- 172.21.15.3:0/1027005479 shutdown_connections 2024-03-29T11:24:20.307 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.322+0000 7f88d0ee4640 1 -- 172.21.15.3:0/1027005479 wait complete. 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.323+0000 7f88d0ee4640 10 monclient: build_initial_monmap 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.323+0000 7f88d0ee4640 10 monclient: monmap: 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.313 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 librados: starting msgr at 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 librados: starting objecter 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 Processor -- start 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 -- start start 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 librados: setting wanted keys 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 1 librados: calling monclient init 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.324+0000 7f88d0ee4640 10 monclient: init 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f88cc0a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:20.314 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f88cc0aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f88cc146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc1d8280 con 0x7f88cc0aab00 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc1d91e0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc140890 con 0x7f88cc146f80 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.325+0000 7f88d0ee4640 10 monclient(hunting): authenticate will time out at 1387.579691s 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88c9d9b640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f88cc0aab00 0x7f88cc140180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47012/0 (socket says 172.21.15.3:47012) 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88c9d9b640 1 -- 172.21.15.3:0/3267996808 learned_addr learned my addr 172.21.15.3:0/3267996808 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter ms_handle_connect 0x7f88cc0aab00 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter ms_handle_connect 0x7f88cc0a98a0 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter ms_handle_connect 0x7f88cc146f80 2024-03-29T11:24:20.315 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424126426 0 0) 0x7f88cc1d8280 con 0x7f88cc0aab00 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ac004eb0 con 0x7f88cc0aab00 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513987328 0 0) 0x7f88cc1d91e0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 10 monclient(hunting): my global_id is 4570 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.326+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88cc1d8280 con 0x7f88cc0a98a0 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307189174 0 0) 0x7f88cc140890 con 0x7f88cc146f80 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:20.316 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88cc1d91e0 con 0x7f88cc146f80 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008068239 0 0) 0x7f88cc1d8280 con 0x7f88cc0a98a0 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88cc140890 con 0x7f88cc0a98a0 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f88bc0017c0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): dump: 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:20.317 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 449336534 0 0) 0x7f88ac004eb0 con 0x7f88cc0aab00 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88cc1d8280 con 0x7f88cc0aab00 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f88c0001bc0 con 0x7f88cc0aab00 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient(hunting): dump: 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.318 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 620682201 0 0) 0x7f88cc140890 con 0x7f88cc0a98a0 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 >> v1:172.21.15.3:6790/0 conn(0x7f88cc146f80 legacy=0x7f88cc13a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 >> v1:172.21.15.3:6789/0 conn(0x7f88cc0aab00 legacy=0x7f88cc140180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 monclient: found mon.b 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:20.319 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc140a20 con 0x7f88cc0a98a0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 5 monclient: authenticate success, global_id 4570 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 10 monclient: _renew_subs 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc13acc0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 10 monclient: _renew_subs 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc13fa50 con 0x7f88cc0a98a0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88d0ee4640 1 librados: init done 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88bc001d50 con 0x7f88cc0a98a0 2024-03-29T11:24:20.320 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f88bc004980 con 0x7f88cc0a98a0 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88baffd640 10 monclient: dump: 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:20.321 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.327+0000 7f88c8d99640 4 set_mon_vals no callback set 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.329+0000 7f88d0ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f88bc01a950 con 0x7f88cc0a98a0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88d0ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88d0ee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b0005080 con 0x7f88cc0a98a0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f88bc0848b0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 10 client.4570.objecter ms_dispatch 0x7f88cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88baffd640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.330+0000 7f88c9d9b640 10 monclient: get_auth_request con 0x7f88ac06dfe0 auth_method 0 2024-03-29T11:24:20.322 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.331+0000 7f88baffd640 10 client.4570.objecter ms_handle_connect 0x7f88ac06dfe0 2024-03-29T11:24:20.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.340+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88bc050190 con 0x7f88cc0a98a0 2024-03-29T11:24:20.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.340+0000 7f88baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:20.323 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.340+0000 7f88baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:20.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.718+0000 7f88d0ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:20.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.718+0000 7f88d0ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:20.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.718+0000 7f88d0ee4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:20.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.718+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88b0005330 con 0x7f88cc0a98a0 2024-03-29T11:24:20.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.719+0000 7f88baffd640 1 -- 172.21.15.3:0/3267996808 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f88bc0520c0 con 0x7f88cc0a98a0 2024-03-29T11:24:20.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.719+0000 7f88baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:20.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.719+0000 7f88baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:20.701 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:20.701 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:20.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.722+0000 7f88d0ee4640 10 librados: watch_flush enter 2024-03-29T11:24:20.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 10 librados: watch_flush exit 2024-03-29T11:24:20.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-29T11:24:20.704 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 10 client.4570.objecter successfully canceled tick 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 >> v1:172.21.15.3:6816/844689376 conn(0x7f88ac06dfe0 legacy=0x7f88ac0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 10 monclient: shutdown 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 >> v1:172.21.15.32:6789/0 conn(0x7f88cc0a98a0 legacy=0x7f88cc1d8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 shutdown_connections 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 >> 172.21.15.3:0/3267996808 conn(0x7f88cc0b3c10 msgr2=0x7f88cc1435b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 shutdown_connections 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.723+0000 7f88d0ee4640 1 -- 172.21.15.3:0/3267996808 wait complete. 2024-03-29T11:24:20.705 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:20.724+0000 7f88d0ee4640 1 librados: shutdown 2024-03-29T11:24:20.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:27.527 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:27.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.708+0000 7f1c152d6640 10 monclient: get_monmap_and_config 2024-03-29T11:24:27.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.708+0000 7f1c152d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:27.690 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.708+0000 7f1c152d6640 10 monclient: monmap: 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:27.709834+0000 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:27.709834+0000 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:27.691 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.709+0000 7f1c152d6640 1 Processor -- start 2024-03-29T11:24:27.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 1 -- start start 2024-03-29T11:24:27.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient: init 2024-03-29T11:24:27.692 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c100b28d0 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c100b2cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.710+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c100b3560 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c152d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100aab60 con 0x7f1c100b28d0 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c152d6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100aae40 con 0x7f1c100b2cb0 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c152d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c100ab120 con 0x7f1c100b3560 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c152d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c152d6640 10 monclient(hunting): authenticate will time out at 1394.965021s 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0effd640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f1c100b28d0 0x7f1c1013bc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33908/0 (socket says 172.21.15.3:33908) 2024-03-29T11:24:27.693 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0effd640 1 -- 172.21.15.3:0/3779650730 learned_addr learned my addr 172.21.15.3:0/3779650730 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:27.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3309502179 0 0) 0x7f1c100aab60 con 0x7f1c100b28d0 2024-03-29T11:24:27.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:27.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:27.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:27.696 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.711+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bfc003520 con 0x7f1c100b28d0 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116596381 0 0) 0x7f1c100ab120 con 0x7f1c100b3560 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c100aab60 con 0x7f1c100b3560 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3951252042 0 0) 0x7f1bfc003520 con 0x7f1c100b28d0 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 >> v1:172.21.15.3:6790/0 conn(0x7f1c100b3560 legacy=0x7f1c101428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 >> v1:172.21.15.32:6789/0 conn(0x7f1c100b2cb0 legacy=0x7f1c10142190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 monclient: found mon.noname-a 2024-03-29T11:24:27.697 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1013c380 con 0x7f1c100b28d0 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c152d6640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c152d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1c00001690 con 0x7f1c100b28d0 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c0dffb640 10 monclient: dump: 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.698 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.712+0000 7f1c152d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c000018b0 con 0x7f1c100b28d0 2024-03-29T11:24:27.699 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c152d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c152d6640 4 set_mon_vals no callback set 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 1 -- 172.21.15.3:0/3779650730 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1c00001c40 con 0x7f1c100b28d0 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c0dffb640 10 monclient: dump: 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.700 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c152d6640 10 monclient: shutdown 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c152d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.713+0000 7f1c152d6640 1 -- 172.21.15.3:0/3779650730 >> v1:172.21.15.3:6789/0 conn(0x7f1c100b28d0 legacy=0x7f1c1013bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.714+0000 7f1c152d6640 1 -- 172.21.15.3:0/3779650730 shutdown_connections 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.714+0000 7f1c152d6640 1 -- 172.21.15.3:0/3779650730 >> 172.21.15.3:0/3779650730 conn(0x7f1c10066a00 msgr2=0x7f1c10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.714+0000 7f1c152d6640 1 -- 172.21.15.3:0/3779650730 shutdown_connections 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.714+0000 7f1c152d6640 1 -- 172.21.15.3:0/3779650730 wait complete. 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.714+0000 7f1c152d6640 10 monclient: build_initial_monmap 2024-03-29T11:24:27.701 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 10 monclient: monmap: 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 librados: starting msgr at 2024-03-29T11:24:27.702 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 librados: starting objecter 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 Processor -- start 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 -- start start 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 librados: setting wanted keys 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 1 librados: calling monclient init 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.715+0000 7f1c152d6640 10 monclient: init 2024-03-29T11:24:27.710 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c100b28d0 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c10144410 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c10136cc0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c1013f510 con 0x7f1c10144410 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c10137150 con 0x7f1c10136cc0 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c101461d0 con 0x7f1c100b28d0 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:27.711 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c152d6640 10 monclient(hunting): authenticate will time out at 1394.970622s 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c0f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f1c10136cc0 0x7f1c101e1f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:51156/0 (socket says 172.21.15.3:51156) 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.716+0000 7f1c0f7fe640 1 -- 172.21.15.3:0/2823262389 learned_addr learned my addr 172.21.15.3:0/2823262389 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter ms_handle_connect 0x7f1c10144410 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter ms_handle_connect 0x7f1c10136cc0 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter ms_handle_connect 0x7f1c100b28d0 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606789453 0 0) 0x7f1c10137150 con 0x7f1c10136cc0 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:27.712 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0004eb0 con 0x7f1c10136cc0 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571255421 0 0) 0x7f1c101461d0 con 0x7f1c100b28d0 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.717+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c10137150 con 0x7f1c100b28d0 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352376592 0 0) 0x7f1c1013f510 con 0x7f1c10144410 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:24:27.713 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c101461d0 con 0x7f1c10144410 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143326257 0 0) 0x7f1bf0004eb0 con 0x7f1c10136cc0 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c1013f510 con 0x7f1c10136cc0 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1bf8001c30 con 0x7f1c10136cc0 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): dump: 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.714 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1234097587 0 0) 0x7f1c101461d0 con 0x7f1c10144410 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf0004eb0 con 0x7f1c10144410 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1576344050 0 0) 0x7f1c10137150 con 0x7f1c100b28d0 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c101461d0 con 0x7f1c100b28d0 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1c08001770 con 0x7f1c10144410 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): dump: 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.715 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1c00001430 con 0x7f1c100b28d0 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient(hunting): dump: 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.716 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 736738462 0 0) 0x7f1c1013f510 con 0x7f1c10136cc0 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 >> v1:172.21.15.3:6790/0 conn(0x7f1c100b28d0 legacy=0x7f1c10144d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 >> v1:172.21.15.3:6789/0 conn(0x7f1c10144410 legacy=0x7f1c1013ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 monclient: found mon.b 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:27.717 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.718+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c101365c0 con 0x7f1c10136cc0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c077fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 5 monclient: authenticate success, global_id 4576 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 10 monclient: _renew_subs 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c101368a0 con 0x7f1c10136cc0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 10 monclient: _renew_subs 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c101e3690 con 0x7f1c10136cc0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c152d6640 1 librados: init done 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bf8003970 con 0x7f1c10136cc0 2024-03-29T11:24:27.718 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.719+0000 7f1c0d7fa640 4 set_mon_vals no callback set 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f1bf8003d80 con 0x7f1c10136cc0 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c077fe640 10 monclient: dump: 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:27.719 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c152d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c152d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c152d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.720+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf4005080 con 0x7f1c10136cc0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.722+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f1bf8003ee0 con 0x7f1c10136cc0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.722+0000 7f1c0effd640 10 monclient: get_auth_request con 0x7f1bf006df90 auth_method 0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7f1bf8084d00 con 0x7f1c10136cc0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 10 client.4576.objecter ms_dispatch 0x7f1c10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-29T11:24:27.720 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.723+0000 7f1c077fe640 10 client.4576.objecter ms_handle_connect 0x7f1bf006df90 2024-03-29T11:24:27.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.728+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bf80505e0 con 0x7f1c10136cc0 2024-03-29T11:24:27.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.728+0000 7f1c077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:27.721 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:27.728+0000 7f1c077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:28.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.122+0000 7f1c152d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:28.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.122+0000 7f1c152d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:28.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.122+0000 7f1c152d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.32:6789/0 2024-03-29T11:24:28.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.122+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf4005330 con 0x7f1c10136cc0 2024-03-29T11:24:28.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.125+0000 7f1c077fe640 1 -- 172.21.15.3:0/2823262389 <== mon.1 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7f1bf8052510 con 0x7f1c10136cc0 2024-03-29T11:24:28.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.125+0000 7f1c077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:28.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.125+0000 7f1c077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:28.107 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:28.107 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.128+0000 7f1c152d6640 10 librados: watch_flush enter 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.128+0000 7f1c152d6640 10 librados: watch_flush exit 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.128+0000 7f1c152d6640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.128+0000 7f1c152d6640 10 client.4576.objecter successfully canceled tick 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 >> v1:172.21.15.3:6816/844689376 conn(0x7f1bf006df90 legacy=0x7f1bf0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 10 monclient: shutdown 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 >> v1:172.21.15.32:6789/0 conn(0x7f1c10136cc0 legacy=0x7f1c101e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 shutdown_connections 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 >> 172.21.15.3:0/2823262389 conn(0x7f1c10066a00 msgr2=0x7f1c1013b760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:28.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 shutdown_connections 2024-03-29T11:24:28.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.129+0000 7f1c152d6640 1 -- 172.21.15.3:0/2823262389 wait complete. 2024-03-29T11:24:28.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:28.130+0000 7f1c152d6640 1 librados: shutdown 2024-03-29T11:24:28.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:34.933 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.102+0000 7fbb03420640 10 monclient: get_monmap_and_config 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.102+0000 7fbb03420640 10 monclient: build_initial_monmap 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.102+0000 7fbb03420640 10 monclient: monmap: 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.084 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:35.103037+0000 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:35.103037+0000 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.102+0000 7fbb03420640 1 Processor -- start 2024-03-29T11:24:35.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.103+0000 7fbb03420640 1 -- start start 2024-03-29T11:24:35.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.103+0000 7fbb03420640 10 monclient: init 2024-03-29T11:24:35.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.104+0000 7fbb03420640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:35.087 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.104+0000 7fbb03420640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.104+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-a con 0x7fbafc0b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-c con 0x7fbafc0b4f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-b con 0x7fbafc0a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0aabd0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0aaeb0 con 0x7fbafc0b4f20 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc0ab190 con 0x7fbafc0a9a90 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:35.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.105+0000 7fbb03420640 10 monclient(hunting): authenticate will time out at 1402.359907s 2024-03-29T11:24:35.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.107+0000 7fbb011bc640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fbafc0b4b40 0x7fbafc13be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:44371/0 (socket says 172.21.15.3:44371) 2024-03-29T11:24:35.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.108+0000 7fbb009bb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbafc0b4f20 0x7fbafc1455f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:39682/0 (socket says 172.21.15.3:39682) 2024-03-29T11:24:35.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.108+0000 7fbb009bb640 1 -- 172.21.15.3:0/562701283 learned_addr learned my addr 172.21.15.3:0/562701283 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814476945 0 0) 0x7fbafc0aabd0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.109+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8003520 con 0x7fbafc0b4b40 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244917360 0 0) 0x7fbae8003520 con 0x7fbafc0b4b40 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 >> v1:172.21.15.3:6790/0 conn(0x7fbafc0a9a90 legacy=0x7fbafc145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 >> v1:172.21.15.32:6789/0 conn(0x7fbafc0b4f20 legacy=0x7fbafc1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 monclient: found mon.noname-a 2024-03-29T11:24:35.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc0b45a0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: _finish_auth 0 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbaf8001690 con 0x7fbafc0b4b40 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: dump: 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.095 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbb03420640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbb03420640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf8001860 con 0x7fbafc0b4b40 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbaf3fff640 1 -- 172.21.15.3:0/562701283 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbaf8001cb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbb03420640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.110+0000 7fbb03420640 4 set_mon_vals no callback set 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbb03420640 10 monclient: shutdown 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbaf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:35.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbaf3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbaf3fff640 10 monclient: dump: 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbb03420640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbb03420640 1 -- 172.21.15.3:0/562701283 >> v1:172.21.15.3:6789/0 conn(0x7fbafc0b4b40 legacy=0x7fbafc13be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbb03420640 1 -- 172.21.15.3:0/562701283 shutdown_connections 2024-03-29T11:24:35.097 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.111+0000 7fbb03420640 1 -- 172.21.15.3:0/562701283 >> 172.21.15.3:0/562701283 conn(0x7fbafc066a00 msgr2=0x7fbafc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:35.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.112+0000 7fbb03420640 1 -- 172.21.15.3:0/562701283 shutdown_connections 2024-03-29T11:24:35.098 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.112+0000 7fbb03420640 1 -- 172.21.15.3:0/562701283 wait complete. 2024-03-29T11:24:35.099 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.112+0000 7fbb03420640 10 monclient: build_initial_monmap 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.112+0000 7fbb03420640 10 monclient: monmap: 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 librados: starting msgr at 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 librados: starting objecter 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 Processor -- start 2024-03-29T11:24:35.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 -- start start 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 librados: setting wanted keys 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 1 librados: calling monclient init 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.113+0000 7fbb03420640 10 monclient: init 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-c con 0x7fbafc0a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-b con 0x7fbafc0b4b40 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 10 monclient(hunting): picked mon.noname-a con 0x7fbafc13d690 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc06ec40 con 0x7fbafc13d690 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc06eeb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.114+0000 7fbb03420640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbafc06f190 con 0x7fbafc0a9a90 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.115+0000 7fbb03420640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:35.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.115+0000 7fbb03420640 10 monclient(hunting): authenticate will time out at 1402.369083s 2024-03-29T11:24:35.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.119+0000 7fbb009bb640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fbafc0b4b40 0x7fbafc06f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:40708/0 (socket says 172.21.15.3:40708) 2024-03-29T11:24:35.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.119+0000 7fbb009bb640 1 -- 172.21.15.3:0/4190892332 learned_addr learned my addr 172.21.15.3:0/4190892332 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fbafc0b4b40 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390538141 0 0) 0x7fbafc06eeb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:35.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae0004eb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fbafc13d690 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter ms_handle_connect 0x7fbafc0a9a90 2024-03-29T11:24:35.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.121+0000 7fbaf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 822200338 0 0) 0x7fbae0004eb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbafc06eeb0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbaec0019d0 con 0x7fbafc0b4b40 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:35.105 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407794956 0 0) 0x7fbafc06f190 con 0x7fbafc0a9a90 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:35.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.122+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae0004eb0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044659408 0 0) 0x7fbae0004eb0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbae0005940 con 0x7fbafc0a9a90 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbaf8001b70 con 0x7fbafc0a9a90 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:35.107 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.108 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:35.108 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:35.108 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:35.108 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 790392613 0 0) 0x7fbae0005940 con 0x7fbafc0a9a90 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 >> v1:172.21.15.32:6789/0 conn(0x7fbafc0b4b40 legacy=0x7fbafc06f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 >> v1:172.21.15.3:6789/0 conn(0x7fbafc13d690 legacy=0x7fbafc131c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 monclient: found mon.c 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc06f470 con 0x7fbafc0a9a90 2024-03-29T11:24:35.109 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbaf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 10 monclient: _renew_subs 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbafc06de80 con 0x7fbafc0a9a90 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:35.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 10 monclient: _renew_subs 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbafc13dcd0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.123+0000 7fbb03420640 1 librados: init done 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbb03420640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbb03420640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbb03420640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbafc067ea0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbaf8001d50 con 0x7fbafc0a9a90 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fbaf8001b70 con 0x7fbafc0a9a90 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf1ffb640 10 monclient: dump: 2024-03-29T11:24:35.111 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.125+0000 7fbaf37fe640 4 set_mon_vals no callback set 2024-03-29T11:24:35.112 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.127+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fbaf801a9a0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.128+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3442886390 0 0) 0x7fbaf80845e0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.128+0000 7fbaf1ffb640 10 client.4601.objecter ms_dispatch 0x7fbafc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:35.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.128+0000 7fbaf1ffb640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:35.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.128+0000 7fbaf1ffb640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:35.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.128+0000 7fbaf1ffb640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-29T11:24:35.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.133+0000 7fbb009bb640 10 monclient: get_auth_request con 0x7fbae006e210 auth_method 0 2024-03-29T11:24:35.115 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.133+0000 7fbaf1ffb640 10 client.4601.objecter ms_handle_connect 0x7fbae006e210 2024-03-29T11:24:35.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.136+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbaf804fec0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.136+0000 7fbaf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:35.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.136+0000 7fbaf1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:35.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.471+0000 7fbb03420640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:35.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.471+0000 7fbb03420640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:35.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.471+0000 7fbb03420640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:35.454 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.471+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbafc067ea0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.473+0000 7fbaf1ffb640 1 -- 172.21.15.3:0/4190892332 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3596950978) 0x7fbaf804fec0 con 0x7fbafc0a9a90 2024-03-29T11:24:35.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.473+0000 7fbaf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:35.455 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.473+0000 7fbaf1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:35.455 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:35.456 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":10,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:14:38.065630+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:14:38.065630+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:35.459 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.477+0000 7fbb03420640 10 librados: watch_flush enter 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.477+0000 7fbb03420640 10 librados: watch_flush exit 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.477+0000 7fbb03420640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.477+0000 7fbb03420640 10 client.4601.objecter successfully canceled tick 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 >> v1:172.21.15.3:6816/844689376 conn(0x7fbae006e210 legacy=0x7fbae00706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 10 monclient: shutdown 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 >> v1:172.21.15.3:6790/0 conn(0x7fbafc0a9a90 legacy=0x7fbafc13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 shutdown_connections 2024-03-29T11:24:35.460 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.478+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 >> 172.21.15.3:0/4190892332 conn(0x7fbafc066a00 msgr2=0x7fbafc0b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:35.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.479+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 shutdown_connections 2024-03-29T11:24:35.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.479+0000 7fbb03420640 1 -- 172.21.15.3:0/4190892332 wait complete. 2024-03-29T11:24:35.461 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:35.479+0000 7fbb03420640 1 librados: shutdown 2024-03-29T11:24:35.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:42.284 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:42.422 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.440+0000 7f3a490f4640 10 monclient: get_monmap_and_config 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.440+0000 7f3a490f4640 10 monclient: build_initial_monmap 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.440+0000 7f3a490f4640 10 monclient: monmap: 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:42.441570+0000 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:42.441570+0000 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:42.423 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.441+0000 7f3a490f4640 1 Processor -- start 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.441+0000 7f3a490f4640 1 -- start start 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.441+0000 7f3a490f4640 10 monclient: init 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-b con 0x7f3a440b4b40 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-a con 0x7f3a440b4f20 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a42d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3a440b4b40 0x7f3a4413be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:51462/0 (socket says 172.21.15.3:51462) 2024-03-29T11:24:42.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-c con 0x7f3a440a9a90 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a42d9d640 1 -- 172.21.15.3:0/3978903413 learned_addr learned my addr 172.21.15.3:0/3978903413 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440aabd0 con 0x7f3a440b4f20 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440aaeb0 con 0x7f3a440a9a90 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440ab190 con 0x7f3a440b4b40 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a490f4640 10 monclient(hunting): authenticate will time out at 1409.696431s 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676622473 0 0) 0x7f3a440ab190 con 0x7f3a440b4b40 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.442+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a2c003520 con 0x7f3a440b4b40 2024-03-29T11:24:42.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.443+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4211256254 0 0) 0x7f3a2c003520 con 0x7f3a440b4b40 2024-03-29T11:24:42.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.443+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 >> v1:172.21.15.32:6789/0 conn(0x7f3a440a9a90 legacy=0x7f3a44145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:42.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.443+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 >> v1:172.21.15.3:6789/0 conn(0x7f3a440b4f20 legacy=0x7f3a441455f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 1 monclient: found mon.noname-b 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440b45a0 con 0x7f3a440b4b40 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a34001670 con 0x7f3a440b4b40 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:42.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: dump: 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a490f4640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a490f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a340018d0 con 0x7f3a440b4b40 2024-03-29T11:24:42.430 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a490f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a490f4640 4 set_mon_vals no callback set 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 1 -- 172.21.15.3:0/3978903413 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a34001d20 con 0x7f3a440b4b40 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.445+0000 7f3a41d9b640 10 monclient: dump: 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:42.431 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.446+0000 7f3a490f4640 10 monclient: shutdown 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.446+0000 7f3a490f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.446+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 >> v1:172.21.15.3:6790/0 conn(0x7f3a440b4b40 legacy=0x7f3a4413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.447+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 shutdown_connections 2024-03-29T11:24:42.432 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.447+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 >> 172.21.15.3:0/3978903413 conn(0x7f3a44066a00 msgr2=0x7f3a44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.447+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 shutdown_connections 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.448+0000 7f3a490f4640 1 -- 172.21.15.3:0/3978903413 wait complete. 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.449+0000 7f3a490f4640 10 monclient: build_initial_monmap 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.450+0000 7f3a490f4640 10 monclient: monmap: 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:42.433 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.450+0000 7f3a490f4640 1 librados: starting msgr at 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.450+0000 7f3a490f4640 1 librados: starting objecter 2024-03-29T11:24:42.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.451+0000 7f3a490f4640 1 Processor -- start 2024-03-29T11:24:42.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.452+0000 7f3a490f4640 1 -- start start 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.452+0000 7f3a490f4640 1 librados: setting wanted keys 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.452+0000 7f3a490f4640 1 librados: calling monclient init 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.452+0000 7f3a490f4640 10 monclient: init 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-c con 0x7f3a440a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-a con 0x7f3a440b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4413dda0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a42d9d640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f3a440a9a90 0x7f3a4413e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:57856/0 (socket says 172.21.15.3:57856) 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a42d9d640 1 -- 172.21.15.3:0/2172659675 learned_addr learned my addr 172.21.15.3:0/2172659675 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a44146570 con 0x7f3a440b4b40 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4413ec00 con 0x7f3a4413dda0 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4412e3f0 con 0x7f3a440a9a90 2024-03-29T11:24:42.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.453+0000 7f3a490f4640 10 monclient(hunting): authenticate will time out at 1409.707818s 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f3a440a9a90 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f3a4413dda0 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457018214 0 0) 0x7f3a4412e3f0 con 0x7f3a440a9a90 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.454+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30004eb0 con 0x7f3a440a9a90 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834466501 0 0) 0x7f3a3c001370 con 0x7f3a4413dda0 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:42.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4412e3f0 con 0x7f3a4413dda0 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3155783818 0 0) 0x7f3a30004eb0 con 0x7f3a440a9a90 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a3c001370 con 0x7f3a440a9a90 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a34001d20 con 0x7f3a440a9a90 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.442 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1221343077 0 0) 0x7f3a3c001370 con 0x7f3a440a9a90 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.455+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 >> v1:172.21.15.32:6789/0 conn(0x7f3a4413dda0 legacy=0x7f3a441d44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 >> v1:172.21.15.3:6789/0 conn(0x7f3a440b4b40 legacy=0x7f3a441d78b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 1 monclient: found mon.c 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a441d7390 con 0x7f3a440a9a90 2024-03-29T11:24:42.443 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 10 monclient: _renew_subs 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a441d7fc0 con 0x7f3a440a9a90 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 10 monclient: _renew_subs 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.456+0000 7f3a490f4640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a441d4dd0 con 0x7f3a440a9a90 2024-03-29T11:24:42.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.457+0000 7f3a490f4640 1 librados: init done 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.458+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a34004b40 con 0x7f3a440a9a90 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.458+0000 7f3a2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.458+0000 7f3a297fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.458+0000 7f3a4159a640 4 set_mon_vals no callback set 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f3a34004e50 con 0x7f3a440a9a90 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:42.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a2b7fe640 10 monclient: dump: 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a297fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.459+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a24005080 con 0x7f3a440a9a90 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.461+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f3a3401ae20 con 0x7f3a440a9a90 2024-03-29T11:24:42.447 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a4259c640 10 monclient: get_auth_request con 0x7f3a3006daa0 auth_method 0 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a2b7fe640 10 client.4607.objecter ms_handle_connect 0x7f3a3006daa0 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7f3a34084cb0 con 0x7f3a440a9a90 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a2b7fe640 10 client.4607.objecter ms_dispatch 0x7f3a44000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a2b7fe640 3 client.4607.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.462+0000 7f3a2b7fe640 3 client.4607.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:24:42.448 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.463+0000 7f3a2b7fe640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:24:42.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.468+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a34050590 con 0x7f3a440a9a90 2024-03-29T11:24:42.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.468+0000 7f3a2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:42.451 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.468+0000 7f3a2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.781+0000 7f3a297fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a297fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a24005330 con 0x7f3a440a9a90 2024-03-29T11:24:42.765 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a2b7fe640 1 -- 172.21.15.3:0/2172659675 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7f3a340524c0 con 0x7f3a440a9a90 2024-03-29T11:24:42.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:42.766 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.782+0000 7f3a2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:42.766 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:42.766 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:42.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.785+0000 7f3a297fa640 10 librados: watch_flush enter 2024-03-29T11:24:42.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.785+0000 7f3a297fa640 10 librados: watch_flush exit 2024-03-29T11:24:42.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.785+0000 7f3a297fa640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:24:42.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.785+0000 7f3a297fa640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:24:42.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 >> v1:172.21.15.3:6816/844689376 conn(0x7f3a3006daa0 legacy=0x7f3a3006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 10 monclient: shutdown 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 >> v1:172.21.15.3:6790/0 conn(0x7f3a440a9a90 legacy=0x7f3a4413e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 shutdown_connections 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 >> 172.21.15.3:0/2172659675 conn(0x7f3a44066a00 msgr2=0x7f3a440b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 shutdown_connections 2024-03-29T11:24:42.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 -- 172.21.15.3:0/2172659675 wait complete. 2024-03-29T11:24:42.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:42.786+0000 7f3a297fa640 1 librados: shutdown 2024-03-29T11:24:42.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:49.588 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:49.737 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.755+0000 7ffb2533d640 10 monclient: get_monmap_and_config 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.755+0000 7ffb2533d640 10 monclient: build_initial_monmap 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.755+0000 7ffb2533d640 10 monclient: monmap: 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:49.756116+0000 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:49.756116+0000 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:49.738 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.740 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.756+0000 7ffb2533d640 1 Processor -- start 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.756+0000 7ffb2533d640 1 -- start start 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.756+0000 7ffb2533d640 10 monclient: init 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.756+0000 7ffb2533d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.756+0000 7ffb2533d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb200a98a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb200aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb20146f80 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:49.741 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb200aed20 con 0x7ffb200aab00 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb200aa8b0 con 0x7ffb200a98a0 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb20146a60 con 0x7ffb20146f80 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb2533d640 10 monclient(hunting): authenticate will time out at 1417.011079s 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.757+0000 7ffb1f7fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ffb20146f80 0x7ffb20151cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:50430/0 (socket says 172.21.15.3:50430) 2024-03-29T11:24:49.742 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.758+0000 7ffb1f7fe640 1 -- 172.21.15.3:0/3459586853 learned_addr learned my addr 172.21.15.3:0/3459586853 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:49.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.758+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838923905 0 0) 0x7ffb20146a60 con 0x7ffb20146f80 2024-03-29T11:24:49.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:24:49.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:49.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:49.743 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:24:49.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb08003520 con 0x7ffb20146f80 2024-03-29T11:24:49.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1365356545 0 0) 0x7ffb08003520 con 0x7ffb20146f80 2024-03-29T11:24:49.744 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.759+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 >> v1:172.21.15.32:6789/0 conn(0x7ffb200a98a0 legacy=0x7ffb200ae610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:49.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 >> v1:172.21.15.3:6789/0 conn(0x7ffb200aab00 legacy=0x7ffb201437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:49.745 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:24:49.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 1 monclient: found mon.noname-b 2024-03-29T11:24:49.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:49.746 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb20143ef0 con 0x7ffb20146f80 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: _finish_auth 0 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb18001a10 con 0x7ffb20146f80 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.763+0000 7ffb1dffb640 10 monclient: dump: 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:49.747 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:49.748 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.765+0000 7ffb2533d640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:24:49.749 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.765+0000 7ffb2533d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb18001be0 con 0x7ffb20146f80 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 1 -- 172.21.15.3:0/3459586853 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb18001ff0 con 0x7ffb20146f80 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb1dffb640 10 monclient: dump: 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:49.750 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb2533d640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb2533d640 4 set_mon_vals no callback set 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.766+0000 7ffb2533d640 10 monclient: shutdown 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.767+0000 7ffb2533d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.767+0000 7ffb2533d640 1 -- 172.21.15.3:0/3459586853 >> v1:172.21.15.3:6790/0 conn(0x7ffb20146f80 legacy=0x7ffb20151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.767+0000 7ffb2533d640 1 -- 172.21.15.3:0/3459586853 shutdown_connections 2024-03-29T11:24:49.751 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.767+0000 7ffb2533d640 1 -- 172.21.15.3:0/3459586853 >> 172.21.15.3:0/3459586853 conn(0x7ffb200b3c10 msgr2=0x7ffb200b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:49.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.770+0000 7ffb2533d640 1 -- 172.21.15.3:0/3459586853 shutdown_connections 2024-03-29T11:24:49.753 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.770+0000 7ffb2533d640 1 -- 172.21.15.3:0/3459586853 wait complete. 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.771+0000 7ffb2533d640 10 monclient: build_initial_monmap 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.771+0000 7ffb2533d640 10 monclient: monmap: 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:49.755 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.772+0000 7ffb2533d640 1 librados: starting msgr at 2024-03-29T11:24:49.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.772+0000 7ffb2533d640 1 librados: starting objecter 2024-03-29T11:24:49.756 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.772+0000 7ffb2533d640 1 Processor -- start 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 1 -- start start 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 1 librados: setting wanted keys 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 1 librados: calling monclient init 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 10 monclient: init 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:49.757 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.773+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-c con 0x7ffb200a98a0 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-a con 0x7ffb200aab00 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 10 monclient(hunting): picked mon.noname-b con 0x7ffb20146f80 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb20148780 con 0x7ffb200aab00 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb20070550 con 0x7ffb20146f80 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2006f610 con 0x7ffb200a98a0 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:49.758 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.774+0000 7ffb2533d640 10 monclient(hunting): authenticate will time out at 1417.028888s 2024-03-29T11:24:49.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.776+0000 7ffb1effd640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7ffb200a98a0 0x7ffb2006ef00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:58142/0 (socket says 172.21.15.3:58142) 2024-03-29T11:24:49.760 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.776+0000 7ffb1effd640 1 -- 172.21.15.3:0/2565675820 learned_addr learned my addr 172.21.15.3:0/2565675820 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 client.?.objecter ms_handle_connect 0x7ffb200a98a0 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061654582 0 0) 0x7ffb2006f610 con 0x7ffb200a98a0 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb200a98a0 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 client.?.objecter ms_handle_connect 0x7ffb20146f80 2024-03-29T11:24:49.768 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286672176 0 0) 0x7ffb20070550 con 0x7ffb20146f80 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.779+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2006f610 con 0x7ffb20146f80 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 client.?.objecter ms_handle_connect 0x7ffb200aab00 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47762738 0 0) 0x7ffb20148780 con 0x7ffb200aab00 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:24:49.769 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb20070550 con 0x7ffb200aab00 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851853878 0 0) 0x7ffb0c004eb0 con 0x7ffb200a98a0 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb20148780 con 0x7ffb200a98a0 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb140018b0 con 0x7ffb200a98a0 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): dump: 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.770 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.780+0000 7ffb077fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1904430794 0 0) 0x7ffb2006f610 con 0x7ffb20146f80 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb0c004eb0 con 0x7ffb20146f80 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2698039885 0 0) 0x7ffb20148780 con 0x7ffb200a98a0 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 >> v1:172.21.15.32:6789/0 conn(0x7ffb20146f80 legacy=0x7ffb201ee690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 >> v1:172.21.15.3:6789/0 conn(0x7ffb200aab00 legacy=0x7ffb201e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:49.771 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 monclient: found mon.c 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2006e160 con 0x7ffb200a98a0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 10 monclient: _renew_subs 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb201e7b70 con 0x7ffb200a98a0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 10 monclient: _renew_subs 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:49.772 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb201efdb0 con 0x7ffb200a98a0 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.781+0000 7ffb2533d640 1 librados: init done 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb14003610 con 0x7ffb200a98a0 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7ffb14003960 con 0x7ffb200a98a0 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb077fe640 10 monclient: dump: 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:49.773 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb1d7fa640 4 set_mon_vals no callback set 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.782+0000 7ffb057fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.784+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7ffb14003ac0 con 0x7ffb200a98a0 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7ffb140848c0 con 0x7ffb200a98a0 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb077fe640 10 client.4613.objecter ms_dispatch 0x7ffb20000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb077fe640 3 client.4613.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:24:49.774 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb077fe640 3 client.4613.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb077fe640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb057fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.785+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2006a400 con 0x7ffb200a98a0 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.786+0000 7ffb1e7fc640 10 monclient: get_auth_request con 0x7ffb0c06dcb0 auth_method 0 2024-03-29T11:24:49.775 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.786+0000 7ffb077fe640 10 client.4613.objecter ms_handle_connect 0x7ffb0c06dcb0 2024-03-29T11:24:49.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.795+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb140501a0 con 0x7ffb200a98a0 2024-03-29T11:24:49.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.795+0000 7ffb077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:49.778 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:49.795+0000 7ffb077fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:50.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.130+0000 7ffb057fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:50.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.130+0000 7ffb057fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:50.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.130+0000 7ffb057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:50.113 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.130+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb201e7080 con 0x7ffb200a98a0 2024-03-29T11:24:50.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.131+0000 7ffb077fe640 1 -- 172.21.15.3:0/2565675820 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7ffb140520d0 con 0x7ffb200a98a0 2024-03-29T11:24:50.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.131+0000 7ffb077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:50.114 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.131+0000 7ffb077fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:50.114 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:50.114 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 10 librados: watch_flush enter 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 10 librados: watch_flush exit 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 >> v1:172.21.15.3:6816/844689376 conn(0x7ffb0c06dcb0 legacy=0x7ffb0c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 10 monclient: shutdown 2024-03-29T11:24:50.117 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:50.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 >> v1:172.21.15.3:6790/0 conn(0x7ffb200a98a0 legacy=0x7ffb2006ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:50.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 shutdown_connections 2024-03-29T11:24:50.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 >> 172.21.15.3:0/2565675820 conn(0x7ffb200b3c10 msgr2=0x7ffb20142d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:50.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 shutdown_connections 2024-03-29T11:24:50.118 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.135+0000 7ffb057fa640 1 -- 172.21.15.3:0/2565675820 wait complete. 2024-03-29T11:24:50.119 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:50.136+0000 7ffb057fa640 1 librados: shutdown 2024-03-29T11:24:50.137 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:56.945 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.101+0000 7f2708464640 10 monclient: get_monmap_and_config 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.101+0000 7f2708464640 10 monclient: build_initial_monmap 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.101+0000 7f2708464640 10 monclient: monmap: 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:24:57.102249+0000 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:24:57.102249+0000 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:57.084 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:24:57.085 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:24:57.085 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.102+0000 7f2708464640 1 Processor -- start 2024-03-29T11:24:57.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.102+0000 7f2708464640 1 -- start start 2024-03-29T11:24:57.085 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.102+0000 7f2708464640 10 monclient: init 2024-03-29T11:24:57.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.103+0000 7f2708464640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:57.086 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.103+0000 7f2708464640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 10 monclient(hunting): picked mon.noname-a con 0x7f270006efb0 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 10 monclient(hunting): picked mon.noname-b con 0x7f27000b0e90 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 10 monclient(hunting): picked mon.noname-c con 0x7f270013eca0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000716a0 con 0x7f270006efb0 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700071ac0 con 0x7f270013eca0 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2700071da0 con 0x7f27000b0e90 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.105+0000 7f2708464640 10 monclient(hunting): authenticate will time out at 1424.359561s 2024-03-29T11:24:57.088 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.106+0000 7f27059ff640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f27000b0e90 0x7f270013bdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:39942/0 (socket says 172.21.15.3:39942) 2024-03-29T11:24:57.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.106+0000 7f27059ff640 1 -- 172.21.15.3:0/3663671097 learned_addr learned my addr 172.21.15.3:0/3663671097 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:57.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237825779 0 0) 0x7f2700071da0 con 0x7f27000b0e90 2024-03-29T11:24:57.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:57.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:57.089 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:57.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:24:57.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26ec003520 con 0x7f27000b0e90 2024-03-29T11:24:57.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715044490 0 0) 0x7f26ec003520 con 0x7f27000b0e90 2024-03-29T11:24:57.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 >> v1:172.21.15.32:6789/0 conn(0x7f270013eca0 legacy=0x7f270014a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:57.090 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.107+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 >> v1:172.21.15.3:6789/0 conn(0x7f270006efb0 legacy=0x7f27000722c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 1 monclient: found mon.noname-b 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270013f0d0 con 0x7f27000b0e90 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f26f0001690 con 0x7f27000b0e90 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:57.091 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: dump: 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f2708464640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:24:57.092 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f2708464640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f0001860 con 0x7f27000b0e90 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 1 -- 172.21.15.3:0/3663671097 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f26f0001cb0 con 0x7f27000b0e90 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.108+0000 7f27051fe640 10 monclient: dump: 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.093 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.109+0000 7f2708464640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.109+0000 7f2708464640 4 set_mon_vals no callback set 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.110+0000 7f2708464640 10 monclient: shutdown 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.110+0000 7f2708464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:57.094 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.110+0000 7f2708464640 1 -- 172.21.15.3:0/3663671097 >> v1:172.21.15.3:6790/0 conn(0x7f27000b0e90 legacy=0x7f270013bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.111+0000 7f2708464640 1 -- 172.21.15.3:0/3663671097 shutdown_connections 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.111+0000 7f2708464640 1 -- 172.21.15.3:0/3663671097 >> 172.21.15.3:0/3663671097 conn(0x7f27000b4c20 msgr2=0x7f27000b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.111+0000 7f2708464640 1 -- 172.21.15.3:0/3663671097 shutdown_connections 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.111+0000 7f2708464640 1 -- 172.21.15.3:0/3663671097 wait complete. 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.112+0000 7f2708464640 10 monclient: build_initial_monmap 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.112+0000 7f2708464640 10 monclient: monmap: 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:57.095 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.112+0000 7f2708464640 1 librados: starting msgr at 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.112+0000 7f2708464640 1 librados: starting objecter 2024-03-29T11:24:57.096 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.112+0000 7f2708464640 1 Processor -- start 2024-03-29T11:24:57.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 -- start start 2024-03-29T11:24:57.100 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 librados: setting wanted keys 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 librados: calling monclient init 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient: init 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient(hunting): picked mon.noname-b con 0x7f270006efb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient(hunting): picked mon.noname-c con 0x7f27000b0e90 addr v1:172.21.15.3:6790/0 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient(hunting): picked mon.noname-a con 0x7f270013eca0 addr v1:172.21.15.3:6789/0 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000ac000 con 0x7f270013eca0 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000ab030 con 0x7f270006efb0 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27000ab1c0 con 0x7f27000b0e90 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.113+0000 7f2708464640 10 monclient(hunting): authenticate will time out at 1424.367666s 2024-03-29T11:24:57.101 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f2706200640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f270006efb0 0x7f27000aa920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:36738/0 (socket says 172.21.15.3:36738) 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f2706200640 1 -- 172.21.15.3:0/855771952 learned_addr learned my addr 172.21.15.3:0/855771952 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 client.?.objecter ms_handle_connect 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494329536 0 0) 0x7f27000ab1c0 con 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e4004eb0 con 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 client.?.objecter ms_handle_connect 0x7f270006efb0 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994812308 0 0) 0x7f26e4004eb0 con 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.114+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27000ab1c0 con 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f26f0001b70 con 0x7f27000b0e90 2024-03-29T11:24:57.102 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): dump: 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859925018 0 0) 0x7f27000ab030 con 0x7f270006efb0 2024-03-29T11:24:57.103 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26e4004eb0 con 0x7f270006efb0 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1707913373 0 0) 0x7f27000ab1c0 con 0x7f27000b0e90 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 >> v1:172.21.15.32:6789/0 conn(0x7f270006efb0 legacy=0x7f27000aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 >> v1:172.21.15.3:6789/0 conn(0x7f270013eca0 legacy=0x7f27001dede0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 monclient: found mon.c 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2700139610 con 0x7f27000b0e90 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:57.104 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f26f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 10 monclient: _renew_subs 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27001398c0 con 0x7f27000b0e90 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 10 monclient: _renew_subs 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2700139240 con 0x7f27000b0e90 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.115+0000 7f2708464640 1 librados: init done 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.116+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26f0001cd0 con 0x7f27000b0e90 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f7fff640 4 set_mon_vals no callback set 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f2708464640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:57.105 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f26f0004d90 con 0x7f27000b0e90 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.117+0000 7f26f6ffd640 10 monclient: dump: 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:24:57.106 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.118+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f26f001ad60 con 0x7f27000b0e90 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f2708464640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f2708464640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270006a400 con 0x7f27000b0e90 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f2706200640 10 monclient: get_auth_request con 0x7f26e406dd60 auth_method 0 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7f26f0084c20 con 0x7f27000b0e90 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 10 client.4619.objecter ms_dispatch 0x7f2700000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 3 client.4619.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 3 client.4619.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-29T11:24:57.108 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.119+0000 7f26f6ffd640 10 client.4619.objecter ms_handle_connect 0x7f26e406dd60 2024-03-29T11:24:57.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.127+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26f0050500 con 0x7f27000b0e90 2024-03-29T11:24:57.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.127+0000 7f26f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:57.110 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.127+0000 7f26f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:57.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.457+0000 7f2708464640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:57.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.457+0000 7f2708464640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:57.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.457+0000 7f2708464640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:24:57.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.457+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27000710e0 con 0x7f27000b0e90 2024-03-29T11:24:57.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.458+0000 7f26f6ffd640 1 -- 172.21.15.3:0/855771952 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7f26f0052430 con 0x7f27000b0e90 2024-03-29T11:24:57.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.458+0000 7f26f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:57.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.458+0000 7f26f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:57.442 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:24:57.442 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.461+0000 7f2708464640 10 librados: watch_flush enter 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.461+0000 7f2708464640 10 librados: watch_flush exit 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.461+0000 7f2708464640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.461+0000 7f2708464640 10 client.4619.objecter successfully canceled tick 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.461+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 >> v1:172.21.15.3:6816/844689376 conn(0x7f26e406dd60 legacy=0x7f26e4070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:57.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 10 monclient: shutdown 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 >> v1:172.21.15.3:6790/0 conn(0x7f27000b0e90 legacy=0x7f2700138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 shutdown_connections 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 >> 172.21.15.3:0/855771952 conn(0x7f27000b4c20 msgr2=0x7f2700139f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 shutdown_connections 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 -- 172.21.15.3:0/855771952 wait complete. 2024-03-29T11:24:57.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:24:57.462+0000 7f2708464640 1 librados: shutdown 2024-03-29T11:24:57.462 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:04.269 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:04.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.437+0000 7fa547ff6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:04.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.437+0000 7fa547ff6640 10 monclient: build_initial_monmap 2024-03-29T11:25:04.421 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.437+0000 7fa547ff6640 10 monclient: monmap: 2024-03-29T11:25:04.421 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:04.421 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:25:04.438610+0000 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:25:04.438610+0000 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:25:04.422 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.439+0000 7fa547ff6640 1 Processor -- start 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.439+0000 7fa547ff6640 1 -- start start 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.439+0000 7fa547ff6640 10 monclient: init 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fa5400a9d00 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5400aaef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:04.424 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5400af110 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540141e60 con 0x7fa5400a9d00 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400b0090 con 0x7fa5400af110 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa540141a30 con 0x7fa5400aaef0 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa547ff6640 10 monclient(hunting): authenticate will time out at 1431.694338s 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa546593640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa5400af110 0x7fa5401498d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:53780/0 (socket says 172.21.15.3:53780) 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.440+0000 7fa546593640 1 -- 172.21.15.3:0/2555704457 learned_addr learned my addr 172.21.15.3:0/2555704457 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2942473798 0 0) 0x7fa5400b0090 con 0x7fa5400af110 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 10 monclient(hunting): _negotiate 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 10 monclient(hunting): my global_id is 4597 2024-03-29T11:25:04.425 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.441+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa528003520 con 0x7fa5400af110 2024-03-29T11:25:04.426 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614546780 0 0) 0x7fa540141a30 con 0x7fa5400aaef0 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient(hunting): _negotiate 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa528003840 con 0x7fa5400aaef0 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3275466078 0 0) 0x7fa528003520 con 0x7fa5400af110 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 >> v1:172.21.15.3:6790/0 conn(0x7fa5400aaef0 legacy=0x7fa5400aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 >> v1:172.21.15.3:6789/0 conn(0x7fa5400a9d00 legacy=0x7fa5400af980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: _finish_hunting 0 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 monclient: found mon.noname-c 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5401422f0 con 0x7fa5400af110 2024-03-29T11:25:04.427 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: _finish_auth 0 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: _check_auth_tickets 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa547ff6640 5 monclient: authenticate success, global_id 4597 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa547ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa538005010 con 0x7fa5400af110 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa544d90640 10 monclient: dump: 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:04.428 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.442+0000 7fa547ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5380051f0 con 0x7fa5400af110 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa547ff6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa547ff6640 4 set_mon_vals no callback set 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 1 -- 172.21.15.3:0/2555704457 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa538005010 con 0x7fa5400af110 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:04.429 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.443+0000 7fa544d90640 10 monclient: dump: 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:04.430 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.444+0000 7fa547ff6640 10 monclient: shutdown 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.444+0000 7fa547ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.444+0000 7fa547ff6640 1 -- 172.21.15.3:0/2555704457 >> v1:172.21.15.32:6789/0 conn(0x7fa5400af110 legacy=0x7fa5401498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.444+0000 7fa547ff6640 1 -- 172.21.15.3:0/2555704457 shutdown_connections 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.444+0000 7fa547ff6640 1 -- 172.21.15.3:0/2555704457 >> 172.21.15.3:0/2555704457 conn(0x7fa5400b3c10 msgr2=0x7fa5400b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.445+0000 7fa547ff6640 1 -- 172.21.15.3:0/2555704457 shutdown_connections 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.445+0000 7fa547ff6640 1 -- 172.21.15.3:0/2555704457 wait complete. 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.446+0000 7fa547ff6640 10 monclient: build_initial_monmap 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.446+0000 7fa547ff6640 10 monclient: monmap: 2024-03-29T11:25:04.434 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 librados: starting msgr at 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 librados: starting objecter 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 Processor -- start 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 -- start start 2024-03-29T11:25:04.435 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 librados: setting wanted keys 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 1 librados: calling monclient init 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.447+0000 7fa547ff6640 10 monclient: init 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-a con 0x7fa5400a9d00 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5400aaef0 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa545d92640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fa5400a9d00 0x7fa5401dba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:38110/0 (socket says 172.21.15.3:38110) 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa545d92640 1 -- 172.21.15.3:0/3386758429 learned_addr learned my addr 172.21.15.3:0/3386758429 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5400af110 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5401cd490 con 0x7fa5400a9d00 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5401db520 con 0x7fa5400af110 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54013ceb0 con 0x7fa5400aaef0 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:04.436 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa547ff6640 10 monclient(hunting): authenticate will time out at 1431.702471s 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa53e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5400a9d00 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa53e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa53e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5400af110 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.448+0000 7fa53e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592174524 0 0) 0x7fa5401cd490 con 0x7fa5400a9d00 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa52c004eb0 con 0x7fa5400a9d00 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747842510 0 0) 0x7fa5401db520 con 0x7fa5400af110 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:04.437 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5401cd490 con 0x7fa5400af110 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747737591 0 0) 0x7fa52c004eb0 con 0x7fa5400a9d00 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5401db520 con 0x7fa5400a9d00 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa5300018c0 con 0x7fa5400a9d00 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.438 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3093431762 0 0) 0x7fa5401cd490 con 0x7fa5400af110 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa52c004eb0 con 0x7fa5400af110 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa538005010 con 0x7fa5400af110 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.449+0000 7fa53e7fc640 10 monclient(hunting): dump: 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.439 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 client.?.objecter ms_handle_connect 0x7fa5400aaef0 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2309657268 0 0) 0x7fa5401db520 con 0x7fa5400a9d00 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 >> v1:172.21.15.3:6790/0 conn(0x7fa5400aaef0 legacy=0x7fa5401cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 >> v1:172.21.15.32:6789/0 conn(0x7fa5400af110 legacy=0x7fa54013a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 1 monclient: found mon.a 2024-03-29T11:25:04.440 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5401cae30 con 0x7fa5400a9d00 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa53e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 10 monclient: _renew_subs 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5401dc150 con 0x7fa5400a9d00 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 10 client.4584.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 10 monclient: _renew_subs 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5401cc310 con 0x7fa5400a9d00 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.450+0000 7fa547ff6640 1 librados: init done 2024-03-29T11:25:04.441 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa530003610 con 0x7fa5400a9d00 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fa5300039e0 con 0x7fa5400a9d00 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53f7fe640 4 set_mon_vals no callback set 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.451+0000 7fa53e7fc640 10 monclient: dump: 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:04.442 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:04.443 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:04.443 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:04.443 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.453+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fa530003b40 con 0x7fa5400a9d00 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.453+0000 7fa547ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.453+0000 7fa547ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.453+0000 7fa547ff6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.453+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520005080 con 0x7fa5400a9d00 2024-03-29T11:25:04.444 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa545591640 10 monclient: get_auth_request con 0x7fa52c06dee0 auth_method 0 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7fa5300849f0 con 0x7fa5400a9d00 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 10 client.4584.objecter ms_dispatch 0x7fa540000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 3 client.4584.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 3 client.4584.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 20 client.4584.objecter dump_active .. 0 homeless 2024-03-29T11:25:04.445 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.455+0000 7fa53e7fc640 10 client.4584.objecter ms_handle_connect 0x7fa52c06dee0 2024-03-29T11:25:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.463+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5300502d0 con 0x7fa5400a9d00 2024-03-29T11:25:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.463+0000 7fa53e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:04.446 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.463+0000 7fa53e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:04.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.840+0000 7fa547ff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:04.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.840+0000 7fa547ff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:04.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.840+0000 7fa547ff6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:04.823 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.840+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa520005330 con 0x7fa5400a9d00 2024-03-29T11:25:04.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.841+0000 7fa53e7fc640 1 -- 172.21.15.3:0/3386758429 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7fa530052200 con 0x7fa5400a9d00 2024-03-29T11:25:04.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.841+0000 7fa53e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:04.824 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.841+0000 7fa53e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:04.825 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:25:04.826 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:04.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 10 librados: watch_flush enter 2024-03-29T11:25:04.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 10 librados: watch_flush exit 2024-03-29T11:25:04.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 20 client.4584.objecter shutdown clearing up homeless session... 2024-03-29T11:25:04.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 10 client.4584.objecter successfully canceled tick 2024-03-29T11:25:04.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 >> v1:172.21.15.3:6816/844689376 conn(0x7fa52c06dee0 legacy=0x7fa52c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 10 monclient: shutdown 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.845+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 >> v1:172.21.15.3:6789/0 conn(0x7fa5400a9d00 legacy=0x7fa5401dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.846+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 shutdown_connections 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.846+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 >> 172.21.15.3:0/3386758429 conn(0x7fa5400b3c10 msgr2=0x7fa5400b2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.846+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 shutdown_connections 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.846+0000 7fa547ff6640 1 -- 172.21.15.3:0/3386758429 wait complete. 2024-03-29T11:25:04.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:04.846+0000 7fa547ff6640 1 librados: shutdown 2024-03-29T11:25:04.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:11.650 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.813+0000 7fae57033640 10 monclient: get_monmap_and_config 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.813+0000 7fae57033640 10 monclient: build_initial_monmap 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.813+0000 7fae57033640 10 monclient: monmap: 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:25:11.815466+0000 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:25:11.815466+0000 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.798 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.814+0000 7fae57033640 1 Processor -- start 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.814+0000 7fae57033640 1 -- start start 2024-03-29T11:25:11.799 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.814+0000 7fae57033640 10 monclient: init 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient(hunting): picked mon.noname-b con 0x7fae500af800 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient(hunting): picked mon.noname-a con 0x7fae500b1cd0 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient(hunting): picked mon.noname-c con 0x7fae50145cb0 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae500b0a00 con 0x7fae500b1cd0 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae500b17b0 con 0x7fae50145cb0 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae500aed60 con 0x7fae500af800 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.815+0000 7fae57033640 10 monclient(hunting): authenticate will time out at 1439.070861s 2024-03-29T11:25:11.805 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.819+0000 7fae56031640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7fae500af800 0x7fae501455a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:40885/0 (socket says 172.21.15.3:40885) 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.819+0000 7fae56031640 1 -- 172.21.15.3:0/3090584240 learned_addr learned my addr 172.21.15.3:0/3090584240 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523545682 0 0) 0x7fae500b17b0 con 0x7fae50145cb0 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 10 monclient(hunting): _negotiate 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:25:11.806 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.820+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae3c003520 con 0x7fae50145cb0 2024-03-29T11:25:11.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2146910107 0 0) 0x7fae3c003520 con 0x7fae50145cb0 2024-03-29T11:25:11.808 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 >> v1:172.21.15.3:6790/0 conn(0x7fae500af800 legacy=0x7fae501455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 >> v1:172.21.15.3:6789/0 conn(0x7fae500b1cd0 legacy=0x7fae5013f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: _finish_hunting 0 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 monclient: found mon.noname-c 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae500aeef0 con 0x7fae50145cb0 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: _finish_auth 0 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: _check_auth_tickets 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae48001a10 con 0x7fae50145cb0 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.821+0000 7fae5502f640 10 monclient: dump: 2024-03-29T11:25:11.809 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.824+0000 7fae57033640 5 monclient: authenticate success, global_id 4603 2024-03-29T11:25:11.810 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.824+0000 7fae57033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae48001b70 con 0x7fae50145cb0 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 1 -- 172.21.15.3:0/3090584240 <== mon.1 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae48001fc0 con 0x7fae50145cb0 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae5502f640 10 monclient: dump: 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.813 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae57033640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.826+0000 7fae57033640 4 set_mon_vals no callback set 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.827+0000 7fae57033640 10 monclient: shutdown 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.827+0000 7fae57033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:11.814 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.827+0000 7fae57033640 1 -- 172.21.15.3:0/3090584240 >> v1:172.21.15.32:6789/0 conn(0x7fae50145cb0 legacy=0x7fae5014d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:11.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.830+0000 7fae57033640 1 -- 172.21.15.3:0/3090584240 shutdown_connections 2024-03-29T11:25:11.815 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.830+0000 7fae57033640 1 -- 172.21.15.3:0/3090584240 >> 172.21.15.3:0/3090584240 conn(0x7fae50066a00 msgr2=0x7fae50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:11.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.833+0000 7fae57033640 1 -- 172.21.15.3:0/3090584240 shutdown_connections 2024-03-29T11:25:11.818 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.833+0000 7fae57033640 1 -- 172.21.15.3:0/3090584240 wait complete. 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.834+0000 7fae57033640 10 monclient: build_initial_monmap 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.834+0000 7fae57033640 10 monclient: monmap: 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:25:11.819 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 librados: starting msgr at 2024-03-29T11:25:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 librados: starting objecter 2024-03-29T11:25:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 Processor -- start 2024-03-29T11:25:11.820 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 -- start start 2024-03-29T11:25:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 librados: setting wanted keys 2024-03-29T11:25:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 1 librados: calling monclient init 2024-03-29T11:25:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.835+0000 7fae57033640 10 monclient: init 2024-03-29T11:25:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.836+0000 7fae57033640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:11.821 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.836+0000 7fae57033640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:11.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.838+0000 7fae57033640 10 monclient(hunting): picked mon.noname-a con 0x7fae500af800 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:11.822 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.838+0000 7fae57033640 10 monclient(hunting): picked mon.noname-c con 0x7fae500b1cd0 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:11.826 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.838+0000 7fae57033640 10 monclient(hunting): picked mon.noname-b con 0x7fae50142d50 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae56031640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7fae500af800 0x7fae501dea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:33389/0 (socket says 172.21.15.3:33389) 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae56031640 1 -- 172.21.15.3:0/3313309608 learned_addr learned my addr 172.21.15.3:0/3313309608 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae501df510 con 0x7fae500af800 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae500af800 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae50071f60 con 0x7fae50142d50 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae501df1a0 con 0x7fae500b1cd0 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae57033640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae57033640 10 monclient(hunting): authenticate will time out at 1439.096902s 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.841+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9710461 0 0) 0x7fae501df510 con 0x7fae500af800 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:11.827 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae34004eb0 con 0x7fae500af800 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae50142d50 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.842+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 961957115 0 0) 0x7fae34004eb0 con 0x7fae500af800 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae501df510 con 0x7fae500af800 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae4c001510 con 0x7fae500af800 2024-03-29T11:25:11.828 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369941984 0 0) 0x7fae50071f60 con 0x7fae50142d50 2024-03-29T11:25:11.829 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.843+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae34004eb0 con 0x7fae50142d50 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae500b1cd0 2024-03-29T11:25:11.830 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626801174 0 0) 0x7fae501df1a0 con 0x7fae500b1cd0 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae50071f60 con 0x7fae500b1cd0 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033508026 0 0) 0x7fae50071f60 con 0x7fae500b1cd0 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.844+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae501df1a0 con 0x7fae500b1cd0 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae400017d0 con 0x7fae500b1cd0 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:11.831 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548214538 0 0) 0x7fae501df1a0 con 0x7fae500b1cd0 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 >> v1:172.21.15.32:6789/0 conn(0x7fae50142d50 legacy=0x7fae5006e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 >> v1:172.21.15.3:6789/0 conn(0x7fae500af800 legacy=0x7fae501dea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 monclient: found mon.c 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:11.832 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae501e8db0 con 0x7fae500b1cd0 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae40001d40 con 0x7fae500b1cd0 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.845+0000 7fae5482e640 4 set_mon_vals no callback set 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7fae40003790 con 0x7fae500b1cd0 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae46ffd640 10 monclient: dump: 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.833 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 10 monclient: _renew_subs 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae501e6d90 con 0x7fae500b1cd0 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:11.834 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 10 monclient: _renew_subs 2024-03-29T11:25:11.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:11.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae501e7220 con 0x7fae500b1cd0 2024-03-29T11:25:11.835 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.846+0000 7fae57033640 1 librados: init done 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.848+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7fae40019760 con 0x7fae500b1cd0 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.848+0000 7fae57033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.848+0000 7fae57033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.848+0000 7fae57033640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.848+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae501df330 con 0x7fae500b1cd0 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.849+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7fae40083660 con 0x7fae500b1cd0 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.849+0000 7fae46ffd640 10 client.4628.objecter ms_dispatch 0x7fae50000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.849+0000 7fae46ffd640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.849+0000 7fae46ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:25:11.836 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.849+0000 7fae46ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:25:11.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.850+0000 7fae56031640 10 monclient: get_auth_request con 0x7fae3406e020 auth_method 0 2024-03-29T11:25:11.837 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.850+0000 7fae46ffd640 10 client.4628.objecter ms_handle_connect 0x7fae3406e020 2024-03-29T11:25:11.843 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.859+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae4004ef40 con 0x7fae500b1cd0 2024-03-29T11:25:11.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.859+0000 7fae46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:11.844 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:11.859+0000 7fae46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:12.200 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.216+0000 7fae57033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:12.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.216+0000 7fae57033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:12.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.216+0000 7fae57033640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:12.201 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.216+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae501df6a0 con 0x7fae500b1cd0 2024-03-29T11:25:12.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.218+0000 7fae46ffd640 1 -- 172.21.15.3:0/3313309608 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7fae40050e70 con 0x7fae500b1cd0 2024-03-29T11:25:12.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.218+0000 7fae46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:12.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.218+0000 7fae46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:12.203 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:25:12.203 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:12.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.222+0000 7fae57033640 10 librados: watch_flush enter 2024-03-29T11:25:12.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.222+0000 7fae57033640 10 librados: watch_flush exit 2024-03-29T11:25:12.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.222+0000 7fae57033640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.222+0000 7fae57033640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 >> v1:172.21.15.3:6816/844689376 conn(0x7fae3406e020 legacy=0x7fae340704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 10 monclient: shutdown 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 >> v1:172.21.15.3:6790/0 conn(0x7fae500b1cd0 legacy=0x7fae5006df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 shutdown_connections 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 >> 172.21.15.3:0/3313309608 conn(0x7fae50066a00 msgr2=0x7fae500b58e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 shutdown_connections 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.223+0000 7fae57033640 1 -- 172.21.15.3:0/3313309608 wait complete. 2024-03-29T11:25:12.208 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:12.224+0000 7fae57033640 1 librados: shutdown 2024-03-29T11:25:12.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:19.035 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:19.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.208+0000 7f75e969d640 10 monclient: get_monmap_and_config 2024-03-29T11:25:19.192 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.208+0000 7f75e969d640 10 monclient: build_initial_monmap 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.208+0000 7f75e969d640 10 monclient: monmap: 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:25:19.209788+0000 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:25:19.209788+0000 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:25:19.193 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.210+0000 7f75e969d640 1 Processor -- start 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.210+0000 7f75e969d640 1 -- start start 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.210+0000 7f75e969d640 10 monclient: init 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-b con 0x7f75dc006e30 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-a con 0x7f75dc035e00 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-c con 0x7f75dc0c9670 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:19.198 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc035970 con 0x7f75dc035e00 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc036280 con 0x7f75dc0c9670 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc035640 con 0x7f75dc006e30 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.211+0000 7f75e969d640 10 monclient(hunting): authenticate will time out at 1446.465618s 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.213+0000 7f75e37fe640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f75dc035e00 0x7f75dc0c8f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:39101/0 (socket says 172.21.15.3:39101) 2024-03-29T11:25:19.199 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.213+0000 7f75e37fe640 1 -- 172.21.15.3:0/3980302530 learned_addr learned my addr 172.21.15.3:0/3980302530 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:19.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245733170 0 0) 0x7f75dc035640 con 0x7f75dc006e30 2024-03-29T11:25:19.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:19.203 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.217+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4003520 con 0x7f75dc006e30 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 <== mon.1 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388072268 0 0) 0x7f75dc036280 con 0x7f75dc0c9670 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.218+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc035640 con 0x7f75dc0c9670 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 <== mon.1 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2149085593 0 0) 0x7f75dc035640 con 0x7f75dc0c9670 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 >> v1:172.21.15.3:6790/0 conn(0x7f75dc006e30 legacy=0x7f75dc0cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 >> v1:172.21.15.3:6789/0 conn(0x7f75dc035e00 legacy=0x7f75dc0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.204 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.32:6789/0 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc034e20 con 0x7f75dc0c9670 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e969d640 5 monclient: authenticate success, global_id 4609 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e969d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 <== mon.1 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75e404f730 con 0x7f75dc0c9670 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: dump: 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:19.205 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e969d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 1 -- 172.21.15.3:0/3980302530 <== mon.1 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e4075f80 con 0x7f75dc0c9670 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e969d640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:19.206 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.219+0000 7f75e969d640 4 set_mon_vals no callback set 2024-03-29T11:25:19.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.220+0000 7f75e969d640 10 monclient: shutdown 2024-03-29T11:25:19.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.220+0000 7f75e969d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:19.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.220+0000 7f75e969d640 1 -- 172.21.15.3:0/3980302530 >> v1:172.21.15.32:6789/0 conn(0x7f75dc0c9670 legacy=0x7f75dc0d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.220+0000 7f75e969d640 1 -- 172.21.15.3:0/3980302530 shutdown_connections 2024-03-29T11:25:19.207 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.220+0000 7f75e969d640 1 -- 172.21.15.3:0/3980302530 >> 172.21.15.3:0/3980302530 conn(0x7f75dc005c00 msgr2=0x7f75dc006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:19.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.224+0000 7f75e969d640 1 -- 172.21.15.3:0/3980302530 shutdown_connections 2024-03-29T11:25:19.210 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.226+0000 7f75e969d640 1 -- 172.21.15.3:0/3980302530 wait complete. 2024-03-29T11:25:19.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.227+0000 7f75e969d640 10 monclient: build_initial_monmap 2024-03-29T11:25:19.211 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.227+0000 7f75e969d640 10 monclient: monmap: 2024-03-29T11:25:19.211 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.227+0000 7f75e969d640 1 librados: starting msgr at 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 1 librados: starting objecter 2024-03-29T11:25:19.212 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 1 Processor -- start 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 1 -- start start 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 1 librados: setting wanted keys 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 1 librados: calling monclient init 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.228+0000 7f75e969d640 10 monclient: init 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-b con 0x7f75dc006e30 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-c con 0x7f75dc035e00 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient(hunting): picked mon.noname-a con 0x7f75dc0c9670 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0ceb80 con 0x7f75dc0c9670 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc1658b0 con 0x7f75dc006e30 2024-03-29T11:25:19.216 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0cd7b0 con 0x7f75dc035e00 2024-03-29T11:25:19.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:19.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.229+0000 7f75e969d640 10 monclient(hunting): authenticate will time out at 1446.483400s 2024-03-29T11:25:19.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.234+0000 7f75e37fe640 1 --1- >> v1:172.21.15.3:6790/0 conn(0x7f75dc035e00 0x7f75dc164a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6790/0 says I am v1:172.21.15.3:48695/0 (socket says 172.21.15.3:48695) 2024-03-29T11:25:19.217 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.234+0000 7f75e37fe640 1 -- 172.21.15.3:0/405196434 learned_addr learned my addr 172.21.15.3:0/405196434 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 client.?.objecter ms_handle_connect 0x7f75dc035e00 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 client.?.objecter ms_handle_connect 0x7f75dc006e30 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540128633 0 0) 0x7f75dc0cd7b0 con 0x7f75dc035e00 2024-03-29T11:25:19.222 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c8004eb0 con 0x7f75dc035e00 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970451340 0 0) 0x7f75c8004eb0 con 0x7f75dc035e00 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75dc0cd7b0 con 0x7f75dc035e00 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75d00018b0 con 0x7f75dc035e00 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): dump: 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.223 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.236+0000 7f75e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 813629779 0 0) 0x7f75dc0cd7b0 con 0x7f75dc035e00 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 >> v1:172.21.15.32:6789/0 conn(0x7f75dc006e30 legacy=0x7f75dc0ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 >> v1:172.21.15.3:6789/0 conn(0x7f75dc0c9670 legacy=0x7f75dc1651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 monclient: found mon.c 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:19.224 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc0cda90 con 0x7f75dc035e00 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 10 monclient: _renew_subs 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc0cdd70 con 0x7f75dc035e00 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 10 monclient: _renew_subs 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc0ce100 con 0x7f75dc035e00 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e969d640 1 librados: init done 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75d0001cb0 con 0x7f75dc035e00 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:19.225 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.237+0000 7f75e1ffb640 4 set_mon_vals no callback set 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.238+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f75d0004c30 con 0x7f75dc035e00 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.238+0000 7f75e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.238+0000 7f75e0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.238+0000 7f75e0ff9640 10 monclient: dump: 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:19.226 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:19.227 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.239+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f75d0004d90 con 0x7f75dc035e00 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.240+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7f75d0084b50 con 0x7f75dc035e00 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.240+0000 7f75e0ff9640 10 client.4634.objecter ms_dispatch 0x7f75e4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.240+0000 7f75e0ff9640 3 client.4634.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.240+0000 7f75e0ff9640 3 client.4634.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.240+0000 7f75e0ff9640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.242+0000 7f75e3fff640 10 monclient: get_auth_request con 0x7f75c806dc70 auth_method 0 2024-03-29T11:25:19.228 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.243+0000 7f75e969d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.243+0000 7f75e969d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.244+0000 7f75e969d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.244+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4005080 con 0x7f75dc035e00 2024-03-29T11:25:19.229 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.244+0000 7f75e0ff9640 10 client.4634.objecter ms_handle_connect 0x7f75c806dc70 2024-03-29T11:25:19.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.252+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75d0050430 con 0x7f75dc035e00 2024-03-29T11:25:19.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.252+0000 7f75e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:19.236 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.252+0000 7f75e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:19.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.593+0000 7f75e969d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:19.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.593+0000 7f75e969d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:19.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.593+0000 7f75e969d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.3:6790/0 2024-03-29T11:25:19.579 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.593+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 --> v1:172.21.15.3:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75c4005330 con 0x7f75dc035e00 2024-03-29T11:25:19.580 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:25:19.580 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:19.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.596+0000 7f75e0ff9640 1 -- 172.21.15.3:0/405196434 <== mon.2 v1:172.21.15.3:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7f75d0052360 con 0x7f75dc035e00 2024-03-29T11:25:19.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.596+0000 7f75e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:19.580 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.596+0000 7f75e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:19.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.599+0000 7f75e969d640 10 librados: watch_flush enter 2024-03-29T11:25:19.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 10 librados: watch_flush exit 2024-03-29T11:25:19.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:25:19.583 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 >> v1:172.21.15.3:6816/844689376 conn(0x7f75c806dc70 legacy=0x7f75c8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 10 monclient: shutdown 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 >> v1:172.21.15.3:6790/0 conn(0x7f75dc035e00 legacy=0x7f75dc164a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 shutdown_connections 2024-03-29T11:25:19.584 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.600+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 >> 172.21.15.3:0/405196434 conn(0x7f75dc005c00 msgr2=0x7f75dc0c6af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:19.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.601+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 shutdown_connections 2024-03-29T11:25:19.585 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.601+0000 7f75e969d640 1 -- 172.21.15.3:0/405196434 wait complete. 2024-03-29T11:25:19.586 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:19.602+0000 7f75e969d640 1 librados: shutdown 2024-03-29T11:25:19.607 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:26.414 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:26.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.567+0000 7f2d96d04640 10 monclient: get_monmap_and_config 2024-03-29T11:25:26.551 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.567+0000 7f2d96d04640 10 monclient: build_initial_monmap 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.567+0000 7f2d96d04640 10 monclient: monmap: 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:25:26.568274+0000 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:25:26.568274+0000 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:25:26.552 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.568+0000 7f2d96d04640 1 Processor -- start 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.568+0000 7f2d96d04640 1 -- start start 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.568+0000 7f2d96d04640 10 monclient: init 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-a con 0x7f2d90072320 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:26.554 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-b con 0x7f2d900ae850 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-c con 0x7f2d900ae0a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900aae40 con 0x7f2d90072320 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d900a9a70 con 0x7f2d900ae0a0 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d901421b0 con 0x7f2d900ae850 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.569+0000 7f2d96d04640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.570+0000 7f2d96d04640 10 monclient(hunting): authenticate will time out at 1453.824012s 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.570+0000 7f2d952a1640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2d900ae0a0 0x7f2d9014d930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.3:53956/0 (socket says 172.21.15.3:53956) 2024-03-29T11:25:26.555 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.570+0000 7f2d952a1640 1 -- 172.21.15.3:0/2220457013 learned_addr learned my addr 172.21.15.3:0/2220457013 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797125469 0 0) 0x7f2d900aae40 con 0x7f2d90072320 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d80003520 con 0x7f2d90072320 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1013766480 0 0) 0x7f2d80003520 con 0x7f2d90072320 2024-03-29T11:25:26.556 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.571+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 >> v1:172.21.15.3:6790/0 conn(0x7f2d900ae850 legacy=0x7f2d900ad990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 >> v1:172.21.15.32:6789/0 conn(0x7f2d900ae0a0 legacy=0x7f2d9014d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d900afae0 con 0x7f2d90072320 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f2d84001690 con 0x7f2d90072320 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.572+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:26.558 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.573+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.573+0000 7f2d8f7fe640 10 monclient: dump: 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.573+0000 7f2d96d04640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.573+0000 7f2d96d04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 <== mon.0 v1:172.21.15.3:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d84001860 con 0x7f2d90072320 2024-03-29T11:25:26.559 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:26.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 1 -- 172.21.15.3:0/2220457013 <== mon.0 v1:172.21.15.3:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f2d84001cb0 con 0x7f2d90072320 2024-03-29T11:25:26.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d96d04640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:26.560 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d96d04640 4 set_mon_vals no callback set 2024-03-29T11:25:26.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:26.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:26.563 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.574+0000 7f2d8f7fe640 10 monclient: dump: 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 10 monclient: shutdown 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 1 -- 172.21.15.3:0/2220457013 >> v1:172.21.15.3:6789/0 conn(0x7f2d90072320 legacy=0x7f2d901426d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:26.564 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 1 -- 172.21.15.3:0/2220457013 shutdown_connections 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 1 -- 172.21.15.3:0/2220457013 >> 172.21.15.3:0/2220457013 conn(0x7f2d900b3c10 msgr2=0x7f2d900b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 1 -- 172.21.15.3:0/2220457013 shutdown_connections 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.575+0000 7f2d96d04640 1 -- 172.21.15.3:0/2220457013 wait complete. 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.576+0000 7f2d96d04640 10 monclient: build_initial_monmap 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.576+0000 7f2d96d04640 10 monclient: monmap: 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:25:26.565 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.576+0000 7f2d96d04640 1 librados: starting msgr at 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.576+0000 7f2d96d04640 1 librados: starting objecter 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.576+0000 7f2d96d04640 1 Processor -- start 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.577+0000 7f2d96d04640 1 -- start start 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.577+0000 7f2d96d04640 1 librados: setting wanted keys 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.577+0000 7f2d96d04640 1 librados: calling monclient init 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.577+0000 7f2d96d04640 10 monclient: init 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-c con 0x7f2d90072320 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-b con 0x7f2d900ae0a0 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient(hunting): picked mon.noname-a con 0x7f2d900ae850 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:26.566 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006f530 con 0x7f2d900ae850 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006f810 con 0x7f2d900ae0a0 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9006faf0 con 0x7f2d90072320 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d96d04640 10 monclient(hunting): authenticate will time out at 1453.832419s 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d952a1640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f2d900ae850 0x7f2d901e6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:42464/0 (socket says 172.21.15.3:42464) 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.578+0000 7f2d952a1640 1 -- 172.21.15.3:0/3138053685 learned_addr learned my addr 172.21.15.3:0/3138053685 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 client.?.objecter ms_handle_connect 0x7f2d900ae850 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741068961 0 0) 0x7f2d9006f530 con 0x7f2d900ae850 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:25:26.567 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d74004eb0 con 0x7f2d900ae850 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2114666777 0 0) 0x7f2d74004eb0 con 0x7f2d900ae850 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d9006f530 con 0x7f2d900ae850 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f2d88001c50 con 0x7f2d900ae850 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): dump: 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:26.568 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:26.569 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:26.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1996972691 0 0) 0x7f2d9006f530 con 0x7f2d900ae850 2024-03-29T11:25:26.569 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.579+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 >> v1:172.21.15.3:6790/0 conn(0x7f2d90072320 legacy=0x7f2d90070260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:26.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 >> v1:172.21.15.32:6789/0 conn(0x7f2d900ae0a0 legacy=0x7f2d9013cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:26.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:26.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 1 monclient: found mon.a 2024-03-29T11:25:26.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:26.571 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d9006ddd0 con 0x7f2d900ae850 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d8d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.580+0000 7f2d96d04640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 10 monclient: _renew_subs 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d900ad270 con 0x7f2d900ae850 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 10 monclient: _renew_subs 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d90070b50 con 0x7f2d900ae850 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.581+0000 7f2d96d04640 1 librados: init done 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d88001ff0 con 0x7f2d900ae850 2024-03-29T11:25:26.572 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f2d88004fb0 con 0x7f2d900ae850 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8d7fa640 10 monclient: dump: 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:26.573 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d8effd640 4 set_mon_vals no callback set 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.582+0000 7f2d7affd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.583+0000 7f2d7affd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.583+0000 7f2d7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.583+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d68005080 con 0x7f2d900ae850 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f2d8801af80 con 0x7f2d900ae850 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7f2d88084f30 con 0x7f2d900ae850 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 10 client.4596.objecter ms_dispatch 0x7f2d90000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 3 client.4596.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 3 client.4596.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d8d7fa640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.584+0000 7f2d94aa0640 10 monclient: get_auth_request con 0x7f2d7406db80 auth_method 0 2024-03-29T11:25:26.574 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.585+0000 7f2d8d7fa640 10 client.4596.objecter ms_handle_connect 0x7f2d7406db80 2024-03-29T11:25:26.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.592+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d88050810 con 0x7f2d900ae850 2024-03-29T11:25:26.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.592+0000 7f2d8d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:26.575 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.592+0000 7f2d8d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.923+0000 7f2d7affd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.923+0000 7f2d7affd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.923+0000 7f2d7affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.923+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d68005330 con 0x7f2d900ae850 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.924+0000 7f2d8d7fa640 1 -- 172.21.15.3:0/3138053685 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7f2d88052740 con 0x7f2d900ae850 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.924+0000 7f2d8d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.924+0000 7f2d8d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:25:26.908 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.927+0000 7f2d7affd640 10 librados: watch_flush enter 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.927+0000 7f2d7affd640 10 librados: watch_flush exit 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.927+0000 7f2d7affd640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.927+0000 7f2d7affd640 10 client.4596.objecter successfully canceled tick 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 >> v1:172.21.15.3:6816/844689376 conn(0x7f2d7406db80 legacy=0x7f2d74070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 10 monclient: shutdown 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 >> v1:172.21.15.3:6789/0 conn(0x7f2d900ae850 legacy=0x7f2d901e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 shutdown_connections 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 >> 172.21.15.3:0/3138053685 conn(0x7f2d900b3c10 msgr2=0x7f2d901ccdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:26.912 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 shutdown_connections 2024-03-29T11:25:26.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 -- 172.21.15.3:0/3138053685 wait complete. 2024-03-29T11:25:26.913 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:26.928+0000 7f2d7affd640 1 librados: shutdown 2024-03-29T11:25:26.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:33.735 DEBUG:teuthology.orchestra.run.smithi003:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.890+0000 7f59fad90640 10 monclient: get_monmap_and_config 2024-03-29T11:25:33.874 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.890+0000 7f59fad90640 10 monclient: build_initial_monmap 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.890+0000 7f59fad90640 10 monclient: monmap: 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:25:33.891456+0000 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:25:33.891456+0000 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-c 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-b 2024-03-29T11:25:33.875 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.892+0000 7f59fad90640 1 Processor -- start 2024-03-29T11:25:33.876 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.893+0000 7f59fad90640 1 -- start start 2024-03-29T11:25:33.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.893+0000 7f59fad90640 10 monclient: init 2024-03-29T11:25:33.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:33.877 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:33.878 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-a con 0x7f59f40b4b40 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-c con 0x7f59f40b4f20 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-b con 0x7f59f40a9a90 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40aabd0 con 0x7f59f40b4b40 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40aaeb0 con 0x7f59f40b4f20 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f40ab190 con 0x7f59f40a9a90 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59fad90640 10 monclient(hunting): authenticate will time out at 1461.148134s 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59f8b2c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f59f40b4b40 0x7f59f413be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:46816/0 (socket says 172.21.15.3:46816) 2024-03-29T11:25:33.879 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.894+0000 7f59f8b2c640 1 -- 172.21.15.3:0/758007727 learned_addr learned my addr 172.21.15.3:0/758007727 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:33.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.2 v1:172.21.15.3:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264819279 0 0) 0x7f59f40ab190 con 0x7f59f40a9a90 2024-03-29T11:25:33.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:33.880 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 --> v1:172.21.15.3:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59dc003520 con 0x7f59f40a9a90 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918214662 0 0) 0x7f59f40aabd0 con 0x7f59f40b4b40 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:33.881 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.896+0000 7f59f37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59f40ab190 con 0x7f59f40b4b40 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.2 v1:172.21.15.3:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1969293724 0 0) 0x7f59dc003520 con 0x7f59f40a9a90 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 >> v1:172.21.15.32:6789/0 conn(0x7f59f40b4f20 legacy=0x7f59f41455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 >> v1:172.21.15.3:6789/0 conn(0x7f59f40b4b40 legacy=0x7f59f413be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 1 monclient: found mon.noname-b 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.897+0000 7f59f37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.3:6790/0 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 --> v1:172.21.15.3:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f40b45a0 con 0x7f59f40a9a90 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:33.884 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.2 v1:172.21.15.3:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f59e4001690 con 0x7f59f40a9a90 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: dump: 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:33.885 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59fad90640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59fad90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.2 v1:172.21.15.3:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59e40018b0 con 0x7f59f40a9a90 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 1 -- 172.21.15.3:0/758007727 <== mon.2 v1:172.21.15.3:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f59e4001c40 con 0x7f59f40a9a90 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59fad90640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59fad90640 4 set_mon_vals no callback set 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.898+0000 7f59f37fe640 10 monclient: dump: 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.886 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 10 monclient: shutdown 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 1 -- 172.21.15.3:0/758007727 >> v1:172.21.15.3:6790/0 conn(0x7f59f40a9a90 legacy=0x7f59f4145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 1 -- 172.21.15.3:0/758007727 shutdown_connections 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 1 -- 172.21.15.3:0/758007727 >> 172.21.15.3:0/758007727 conn(0x7f59f4066a00 msgr2=0x7f59f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 1 -- 172.21.15.3:0/758007727 shutdown_connections 2024-03-29T11:25:33.887 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.899+0000 7f59fad90640 1 -- 172.21.15.3:0/758007727 wait complete. 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.900+0000 7f59fad90640 10 monclient: build_initial_monmap 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.900+0000 7f59fad90640 10 monclient: monmap: 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:epoch 0 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:last_changed 0.000000 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:created 0.000000 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.noname-a 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.noname-b 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.noname-c 2024-03-29T11:25:33.893 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 librados: starting msgr at 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 librados: starting objecter 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 Processor -- start 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 -- start start 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 librados: setting wanted keys 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 1 librados: calling monclient init 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.901+0000 7f59fad90640 10 monclient: init 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-a con 0x7f59f40a9a90 addr v1:172.21.15.3:6789/0 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-c con 0x7f59f40b4b40 addr v1:172.21.15.3:6790/0 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient(hunting): picked mon.noname-b con 0x7f59f4145460 addr v1:172.21.15.32:6789/0 2024-03-29T11:25:33.894 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 1 -- --> v1:172.21.15.3:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f413b470 con 0x7f59f40a9a90 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f413f890 con 0x7f59f4145460 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 1 -- --> v1:172.21.15.3:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59f4136810 con 0x7f59f40b4b40 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.902+0000 7f59fad90640 10 monclient(hunting): authenticate will time out at 1461.156678s 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f8b2c640 1 --1- >> v1:172.21.15.3:6789/0 conn(0x7f59f40a9a90 0x7f59f413ead0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.3:6789/0 says I am v1:172.21.15.3:47330/0 (socket says 172.21.15.3:47330) 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f8b2c640 1 -- 172.21.15.3:0/590448793 learned_addr learned my addr 172.21.15.3:0/590448793 (peer_addr_for_me v1:172.21.15.3:0/0) 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 client.?.objecter ms_handle_connect 0x7f59f40a9a90 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983440345 0 0) 0x7f59f40ab190 con 0x7f59f40a9a90 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:25:33.895 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.904+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0004eb0 con 0x7f59f40a9a90 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1653159497 0 0) 0x7f59e0004eb0 con 0x7f59f40a9a90 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59f40ab190 con 0x7f59f40a9a90 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f59ec001af0 con 0x7f59f40a9a90 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 10 monclient(hunting): dump: 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:33.896 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1972203535 0 0) 0x7f59f40ab190 con 0x7f59f40a9a90 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 >> v1:172.21.15.3:6790/0 conn(0x7f59f40b4b40 legacy=0x7f59f4136f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.905+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 >> v1:172.21.15.32:6789/0 conn(0x7f59f4145460 legacy=0x7f59f41e2090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 1 monclient: found mon.a 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59f4136af0 con 0x7f59f40a9a90 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 5 monclient: authenticate success, global_id 4602 2024-03-29T11:25:33.897 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 10 monclient: _renew_subs 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f413e5b0 con 0x7f59f40a9a90 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 10 monclient: _renew_subs 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59f413da60 con 0x7f59f40a9a90 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59fad90640 1 librados: init done 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59ec001d50 con 0x7f59f40a9a90 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1074332929 0 0) 0x7f59ec004d80 con 0x7f59f40a9a90 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.906+0000 7f59f17fa640 10 monclient: dump: 2024-03-29T11:25:33.898 INFO:teuthology.misc.health.smithi003.stderr:epoch 1 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:fsid bc4c2e6b-f00f-48b8-a384-90ebb3c19722 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:last_changed 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:created 2024-03-29T11:13:59.540928+0000 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:election_strategy: 1 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:0: v1:172.21.15.3:6789/0 mon.a 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:1: v1:172.21.15.32:6789/0 mon.b 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:2: v1:172.21.15.3:6790/0 mon.c 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr: 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.908+0000 7f59f2ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:33.899 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.908+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525380420 0 0) 0x7f59ec004ee0 con 0x7f59f40a9a90 2024-03-29T11:25:33.900 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.909+0000 7f59fad90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1662669797 0 0) 0x7f59ec084df0 con 0x7f59f40a9a90 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f17fa640 10 client.4602.objecter ms_dispatch 0x7f59f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f17fa640 3 client.4602.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f17fa640 3 client.4602.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f17fa640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59fad90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59fad90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d4005080 con 0x7f59f40a9a90 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.910+0000 7f59f3fff640 10 monclient: get_auth_request con 0x7f59e006dc40 auth_method 0 2024-03-29T11:25:33.901 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.911+0000 7f59f17fa640 10 client.4602.objecter ms_handle_connect 0x7f59e006dc40 2024-03-29T11:25:33.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.920+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59ec050500 con 0x7f59f40a9a90 2024-03-29T11:25:33.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.920+0000 7f59f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:33.904 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:33.920+0000 7f59f17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:34.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.255+0000 7f59fad90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:34.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.255+0000 7f59fad90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:34.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.255+0000 7f59fad90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.3:6789/0 2024-03-29T11:25:34.239 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.255+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 --> v1:172.21.15.3:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d4005330 con 0x7f59f40a9a90 2024-03-29T11:25:34.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.256+0000 7f59f17fa640 1 -- 172.21.15.3:0/590448793 <== mon.0 v1:172.21.15.3:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1667613235) 0x7f59ec052600 con 0x7f59f40a9a90 2024-03-29T11:25:34.240 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.256+0000 7f59f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:34.241 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.256+0000 7f59f17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:34.241 INFO:teuthology.misc.health.smithi003.stdout: 2024-03-29T11:25:34.241 INFO:teuthology.misc.health.smithi003.stdout:{"epoch":11,"fsid":"bc4c2e6b-f00f-48b8-a384-90ebb3c19722","created":"2024-03-29T11:14:35.852650+0000","modified":"2024-03-29T11:24:40.780280+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:24:40.780280+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"813cc1eb-fd67-4adc-9120-a9ef762b40d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"1cecc8b9-c752-4e32-9cd2-943602908be5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5db1ce7-9b4e-4d54-994d-dfc8d13f7f70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"2a3ca543-5fae-4475-bd72-a4bb45c9d0aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e3918c69-a425-45af-aa06-652fea1f9efa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dcc5177c-2cec-4390-899d-ddee6450a1e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d2642940-19a3-43e0-8402-e924f9fbe3b4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"9a2d4629-bd65-420b-855e-3ca59aa5c06f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 10 librados: watch_flush enter 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 10 librados: watch_flush exit 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 10 client.4602.objecter successfully canceled tick 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 >> v1:172.21.15.3:6816/844689376 conn(0x7f59e006dc40 legacy=0x7f59e0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 10 monclient: shutdown 2024-03-29T11:25:34.245 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.260+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 >> v1:172.21.15.3:6789/0 conn(0x7f59f40a9a90 legacy=0x7f59f413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.261+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 shutdown_connections 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.261+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 >> 172.21.15.3:0/590448793 conn(0x7f59f4066a00 msgr2=0x7f59f40b8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.261+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 shutdown_connections 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.261+0000 7f59fad90640 1 -- 172.21.15.3:0/590448793 wait complete. 2024-03-29T11:25:34.246 INFO:teuthology.misc.health.smithi003.stderr:2024-03-29T11:25:34.261+0000 7f59fad90640 1 librados: shutdown 2024-03-29T11:25:34.262 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:34.263 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:25:34.265 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:25:34.265 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:25:34.265 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.299 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:25:34.300 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:25:34.300 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.376 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:25:34.376 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:25:34.376 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.448 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:25:34.448 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:25:34.448 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.524 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:25:34.524 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:25:34.524 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.555 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:25:34.556 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:25:34.556 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.624 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:25:34.625 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:25:34.625 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.702 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:25:34.702 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:25:34.702 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.778 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:25:34.778 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:25:34.779 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:25:34.779 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.803 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:25:34.803 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:25:34.804 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.842 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:25:34.843 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:25:34.843 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:25:34.843 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.855 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:25:34.856 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:25:34.856 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.908 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:25:34.908 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:25:34.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:25:34.918 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:25:34.918 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:25:34.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:25:34.981 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:25:34.981 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:25:36.802 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:25:36.802 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:25:36.898 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:25:36.898 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:25:36.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:25:36.958 DEBUG:teuthology.orchestra.run.smithi003:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:25:37.025 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:25:37.025 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:25:40.112 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:25:40.112 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:25:40.207 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:25:40.207 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:25:40.248 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:25:40.248 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:25:40.339 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:25:40.341 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/data/mon.a.tgz 2024-03-29T11:25:40.342 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-29T11:25:40.360 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.vAsn8JmdX5 2024-03-29T11:25:40.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.vAsn8JmdX5 2024-03-29T11:25:40.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.vAsn8JmdX5 2024-03-29T11:25:40.582 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.vAsn8JmdX5 is 126KB 2024-03-29T11:25:40.606 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.vAsn8JmdX5 2024-03-29T11:25:40.620 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/data/mon.c.tgz 2024-03-29T11:25:40.620 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-29T11:25:40.675 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.jmkEu4mM7b 2024-03-29T11:25:40.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.jmkEu4mM7b 2024-03-29T11:25:40.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.jmkEu4mM7b 2024-03-29T11:25:40.849 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.jmkEu4mM7b is 126KB 2024-03-29T11:25:40.900 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.jmkEu4mM7b 2024-03-29T11:25:40.914 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/data/mon.b.tgz 2024-03-29T11:25:40.915 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-29T11:25:40.932 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.VOdZuQzEwR 2024-03-29T11:25:40.933 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.VOdZuQzEwR 2024-03-29T11:25:41.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.VOdZuQzEwR 2024-03-29T11:25:41.122 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.VOdZuQzEwR is 126KB 2024-03-29T11:25:41.145 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.VOdZuQzEwR 2024-03-29T11:25:41.162 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:25:41.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:25:41.165 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:25:41.244 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:25:41.282 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:25:41.283 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:25:41.285 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi003/crash 2024-03-29T11:25:41.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:25:41.313 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi032/crash 2024-03-29T11:25:41.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:25:41.349 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:25:41.350 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:25:41.357 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:25:41.419 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39764.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38317.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39253.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38317.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39764.log.gz 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41342.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38317.log.gz 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39959.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39253.log.gz 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40367.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38473.log 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41342.log.gz 2024-03-29T11:25:41.420 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40913.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39959.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39175.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40367.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38120.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38473.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41225.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40913.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38120.log: 86.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.39175.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39842.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38120.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39722.log 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41225.log.gz 2024-03-29T11:25:41.421 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38902.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40076.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39842.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40796.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39722.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38551.log 86.7% 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38902.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38512.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40076.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40037.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40796.log.gz 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39370.log 2024-03-29T11:25:41.422 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38551.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38512.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39683.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40037.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39370.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38668.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40170.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39683.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-03-29T11:25:41.423 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38668.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40874.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40170.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38707.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40874.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39565.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40211.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40874.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39565.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38707.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40211.log: gzip 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39565.log.gz -5 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38590.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40601.log 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40211.log.gz 2024-03-29T11:25:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40115.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40601.log: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38590.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38434.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39803.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40601.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40289.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40115.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38434.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38003.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39803.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40289.log.gz 2024-03-29T11:25:41.425 INFO:teuthology.orchestra.run.smithi003.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38081.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41147.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38003.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41420.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38081.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39331.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41147.log: /var/log/ceph/ceph-client.admin.41420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39881.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41147.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41030.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41420.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39644.log 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39331.log.gz 2024-03-29T11:25:41.426 INFO:teuthology.orchestra.run.smithi003.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39881.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41030.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39526.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41030.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38198.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39644.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38042.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39487.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39526.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38198.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39920.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38042.log.gz 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40718.log 2024-03-29T11:25:41.427 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39487.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39920.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41069.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40562.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40718.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40679.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39097.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41069.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38863.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40679.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41381.log 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40679.log.gz 2024-03-29T11:25:41.428 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41381.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38863.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38159.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41381.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38824.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38941.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38159.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40328.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38824.log.gz 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39292.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38980.log 2024-03-29T11:25:41.429 INFO:teuthology.orchestra.run.smithi003.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38941.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41108.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39292.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39604.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38980.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38278.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39604.log: gzip -5 --verbose -- 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41108.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.37964.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38278.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39604.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.37964.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38278.log.gz 2024-03-29T11:25:41.430 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38356.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37964.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34667.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40484.log: gzip -5 --verbose -- 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38356.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr: /var/log/ceph/ceph-client.admin.39019.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34667.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39019.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34667.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40757.log: gzip -5 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39019.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40835.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-osd.0.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40835.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:25:41.431 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.41303.log: 86.4%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40835.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39214.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38239.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38239.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39214.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38239.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40640.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38785.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.34564.log: /var/log/ceph/ceph-client.admin.40640.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40640.log.gz 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28465.log 2024-03-29T11:25:41.432 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38785.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38785.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39998.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39409.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph.tmp-client.admin.28465.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28465.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38629.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39998.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.39409.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39409.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34273.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38395.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38629.log: /var/log/ceph/ceph-client.admin.34273.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38629.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40445.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34273.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.38395.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38395.log.gz 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:25:41.433 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/ceph-client.admin.40445.log: /var/log/ceph/ceph-mon.c.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40445.log.gz 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34526.log 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34486.log 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34646.log 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34686.log 2024-03-29T11:25:41.440 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34526.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34646.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34686.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34646.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34686.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34566.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34446.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34566.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34726.log 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34566.log.gz 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:25:41.441 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34446.log.gz 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34726.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34726.log.gz 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34606.log 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.b.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.34606.log: 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34606.log.gz 2024-03-29T11:25:41.442 INFO:teuthology.orchestra.run.smithi032.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:25:41.458 INFO:teuthology.orchestra.run.smithi003.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:25:53.017 INFO:teuthology.orchestra.run.smithi003.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:25:53.247 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:25:53.269 INFO:teuthology.orchestra.run.smithi003.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:25:53.278 INFO:teuthology.orchestra.run.smithi032.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:25:53.328 INFO:teuthology.orchestra.run.smithi032.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:25:57.086 INFO:teuthology.orchestra.run.smithi003.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:25:57.399 INFO:teuthology.orchestra.run.smithi003.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:25:57.600 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:26:32.246 INFO:teuthology.orchestra.run.smithi003.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:26:35.267 INFO:teuthology.orchestra.run.smithi003.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:26:35.269 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-29T11:26:35.269 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m53.898s 2024-03-29T11:26:35.269 INFO:teuthology.orchestra.run.smithi003.stderr:user 2m34.623s 2024-03-29T11:26:35.269 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m5.061s 2024-03-29T11:26:56.249 INFO:teuthology.orchestra.run.smithi032.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:26:56.250 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-29T11:26:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:real 1m14.840s 2024-03-29T11:26:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:user 2m2.220s 2024-03-29T11:26:56.251 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m4.155s 2024-03-29T11:26:56.251 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:26:56.251 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi003/log 2024-03-29T11:26:56.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:27:18.124 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi032/log 2024-03-29T11:27:18.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:27:36.064 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:27:36.278 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=98670ecf5ce8410586449781c2d12b3d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:27:36.282 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:27:36.290 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:27:36.317 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:27:36.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:27:36.336 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:27:36.425 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:27:36.433 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:27:36.433 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:27:36.436 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:27:36.449 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:27:36.452 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:27:36.454 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:27:36.454 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-29T11:27:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 41 +458us[ +458us] +/- 87ms 2024-03-29T11:27:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -124us[ -123us] +/- 50ms 2024-03-29T11:27:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv03.front.sepia.ceph.com 3 6 377 42 -219us[ -218us] +/- 81ms 2024-03-29T11:27:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +449us[ +449us] +/- 87ms 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -117us[ -117us] +/- 50ms 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv03.front.sepia.ceph.com 3 6 377 42 -214us[ -212us] +/- 81ms 2024-03-29T11:27:36.458 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:27:36.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:27:36.469 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:27:36.470 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:27:36.479 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:27:36.566 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:27:36.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:27:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:27:36.668 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:27:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:27:36.701 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:27:36.711 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:27:36.721 INFO:teuthology.task.internal:Duration was 1226.970095 seconds 2024-03-29T11:27:36.721 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:27:36.730 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:27:36.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:27:36.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:27:36.775 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:27:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:27:37.189 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:27:37.189 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-29T11:27:37.189 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:27:37.252 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-03-29T11:27:37.252 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:27:37.283 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:27:37.284 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:27:37.302 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:27:37.360 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:27:37.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:27:37.381 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:27:37.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:27:37.520 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:27:37.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:27:37.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:27:37.577 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:27:37.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:27:37.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:27:37.619 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-29T11:27:37.647 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-03-29T11:27:37.690 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:27:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:27:37.706 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:27:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:27:37.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:27:37.758 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:27:37.759 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi003 2024-03-29T11:27:37.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:27:37.799 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680/remote/smithi032 2024-03-29T11:27:37.800 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:27:37.836 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:27:37.836 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:27:37.839 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:27:37.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:27:37.894 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:27:37.894 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:27:37.904 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:27:37.904 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:27:37.906 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:27:37.918 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:27 /home/ubuntu/cephtest 2024-03-29T11:27:37.939 INFO:teuthology.orchestra.run.smithi032.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:27 /home/ubuntu/cephtest 2024-03-29T11:27:37.941 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:27:37.952 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:27:37.979 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:27:38.010 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-29T11:27:38.010 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:27:38.034 DEBUG:teuthology.nuke:shortname: smithi032 2024-03-29T11:27:38.035 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:27:38.039 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:59:02.104971', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCErejqCMDIglx+W5lAHopXR137IR6ag7yNhJsmQPKTHquPBzHI5AkhyWOyvIJqKqh/X9ekU9S3OZ6seNYT841M='} 2024-03-29T11:27:38.062 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622680', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:59:02.106100', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPRTi96wnTvLKRcksy5oYbWtPhrckGCVWAYOJ5Z/+oJPF+AQexN0e4F3hF+M1K7g2ueVfNWA1GoFJluCEqjwyU='} 2024-03-29T11:27:38.096 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-29T11:27:38.097 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:27:38.121 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-29T11:27:38.121 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:38.140 INFO:teuthology.orchestra.console.smithi032:Power off 2024-03-29T11:27:38.141 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:27:38.155 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-29T11:27:38.172 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-03-29T11:27:38.172 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:38.199 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-29T11:27:42.157 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:42.201 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:42.268 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-29T11:27:42.312 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-29T11:27:46.271 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:46.313 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:27:46.383 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-29T11:27:46.423 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-03-29T11:27:46.485 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-29T11:27:46.524 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-03-29T11:27:46.629 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1226.9700949192047 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=98670ecf5ce8410586449781c2d12b3d status: fail success: false 2024-03-29T11:27:46.630 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:27:46.706 INFO:teuthology.run:FAIL